Parameter Dynamics of Online Machine Learning and Test-time Adaptation
Authors: Jae-Hong Lee
Venue: NeurIPS 2025 poster ()
Keywords: Online machine learning
Test-time adaptation
Probabilistic parameter dynamics
Bayesian deep learning
Primary area: general_machine_learning

Abstract: Pre-trained models based on deep neural networks hold strong potential for cross-domain adaptability. However, this potential is often impeded in online machine learning (OML) settings, where the breakdown of the independent and identically distributed (i.i.d.) assumption leads to unstable adaptation. While recent advances in test-time adaptation (TTA) have addressed aspects of this challenge under unsupervised learning, most existing methods focus exclusively on unsupervised objectives and overlook the risks posed by non-i.i.d. environments and the resulting dynamics of model parameters. In this work, we present a probabilistic framework that models the adaptation process using stochastic differential equations, enabling a principled analysis of parameter distribution dynamics over time. Within this framework, we find that the log-variance of the parameter transition distribution aligns closely with an inverse-gamma distribution under stable and high-performing adaptation conditions. Motivated by this insight, we propose Structured Inverse-Gamma Model Alignment (SIGMA), a novel algorithm that dynamically regulates parameter evolution to preserve inverse-gamma alignment throughout adaptation. Extensive experiments across diverse models, datasets, and adaptation scenarios show that SIGMA consistently enhances the performance of state-of-the-art TTA methods, highlighting the critical role of parameter dynamics in ensuring robust adaptation.

Parameter Dynamics of Online Machine Learning and Test-time Adaptation
Jae-Hong Lee Division of Language & Al Hankuk University Foreign Studies Seoul, Republic of Korea jih kr
Abstract
Pre-trained models based on deep neural networks hold strong potential for cross- domain adaptability. However, this potential is often impeded in online machine learning (OML) settings, where the breakdown of the independent and identically distributed (i.i.d.) assumption leads unstable adaptation. While recent advances in test-time adaptation (TTA) have addressed aspects of this challenge under unsupervised learning, most existing methods focus exclusively unsupervised objectives and overlook the risks posed by non-i.i.d. environments and the resulting dynamics of model parameters. In this work, we present a probabilistic framework that models the adaptation process using stochastic differential equations, enabling principled analysis of parameter distribution dynamies over time. Within this framework, we find that the log-variance of the parameter transition distribution aligns closely with inverse inverse-gamma distribution under stable and high- sergopeefoniin adaptation conditions. Motivated by this insight, we propose Structured Inverse- Gamma Model Alignment (SIGMA), a novel algorithm that dynamically regulates parameter evolution to preserve inverse-gamma alignment throughout adaptation. Extensive experiments across diverse models, datasets, and adaptation scenarios show that SIGMA consistently enhances the performance of state-of-the--e TTA methods, highlighting the critical role of parameter dynamics ensuring robust adaptation.
1 Introduction
The rapid advancement of deep neural networks (DNNs) has given rise to powerful pre-trained models capable of generalizing across wide range of domains [2
4. Despite their versatility, deploying these models in dynamic, resource-constrained environments remains challenging. This difficulty stems from their increasing computational demands and, more fundamentally, from the breakdown of the independent and identically distributed (i.i.d)) assumption in real-world online learning scenarios [16] [50]. Online machine learning (OML) [5] provides a paradigm for addressing such challenges by enabling models to adapt incrementally to non-stationary data streams, where full retraining on large, static datasets is impractical 18 44
In OML settings, data arrive sequentially from various domains, inherently violating the i.i.d. assumption that underpins conventional DNN training. This sequential domain shift often induces a severe degradation in model performance and Consequently, critical goal is to develop robust online adaptation methods to prevent collapse while preserving the adaptability under non-i.i.d. conditions.
generalization ability (21] 27
19 60 9
Test-time adaptation (TTA) has recently emerged as promising direction, combining the challenges of OML with unsupervised learning. TTA aims to adapt pre-trained models to unlabeled test data that arrive sequentially during deployment. The early TTA approach focused on unsupervised learning
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

objectives. For example, TENT [6]] replaces cross-entropy with entropy minimization over model predictions. While such methods ofter improvements in early stages, they are prone to collapse when faced with extended domain shifts 6 10
66 51 Recent efforts have addressed this instability by introducing cample-filtering strategies that downweight or discard high-uncertainty predictions during Despite their empirical success, these methods remain narrowly focused on unsupervised objectives and do not explicitly address the fundamental instability introduced by the non-i.i.d. nature of the data and the evolving dynamics of model parameters.
62
adaptation 45 46
29 43
In this work, we take a fundamentally different perspective. Rather than focusing solely on the loss function, we develop probabilistic framework that explicitly models the dynamics of model parameters during adaptation. Our framework leverages stochastic differential equation (SDE) approximations of stochastic gradient descent (SGD) [3 35], treating SGD as continuous-time stochastic process. By discretizing the SDE, we derive a transition distribution that represents the evolution of the parameter uncertainty over time during the adaptation process (Section (3). Within this framework, we focus on the logarithmic variation (log-variance) of the transition distribution as diagnostic signal. Empirical analysis under both i.i.d. and non-i.idd conditions, in both supervised (SL) and unsupervised (USL) settings, leads to three key findings (Section 4 (1) The log-variance distribution follows inverse-gamma (IG) distribution in stable, high-performing adaptation scenarios. (2) Deviations from the IG distribution are strongly predictive of performance degradation. (3) State-of-the- TTA methods tend to implicitly promote IG alignment, suggesting an unintentional but beneficial form of regularization.
These findings motivate our core hypothesis: explicitly maintaining IG alignment in the log-variance dynamics critical for stable and effective adaptation in non-i.i.d. environments. To realize this principle, we introduce the Structured Inverse-Gamma Model Alignment (SIGMA) algorithm. SIGMA dynamically estimates an appropriate IG distribution using derivative- optimization and adjusts the parameter update trajectory to preserve alignment throughout online adaptation. We validate SIGMA through extensive experiments across multiple models, datasets, and adaptation scenarios. Our results show that SIGMA consistently improves the performance of pate-of-the-an TTA methods, supporting our central claim: alignment of parameter dynamics via IG regularization offers robust and principled foundation under online adaptation settings.
2 Preliminaries
2.1 Test-Time Adaptation
TTA addresses the challenge of adapting well-trained model, pretrained on large-scale labeled source data, to stream of unlabeled test data from evolving domains. The objective is to adapt the model in real time to each new domain sample. Performance is measured by the average error rate (AER) across the test stream. TTA methods, developed from domain adaptation and continual learning research [10 62, 66 511 aim to handle changing data distributions. Recently, filtering strategies have been introduced to exclude or downweight uncertain predictions, improving robustness in multi-domain and non-i.i.d. environments [45
While most existing TTA methods focus on unsupervised loss design, our main contribution directly models and analyzes parameter changes during adaptation. Specifically, we propose a probabilistic framework that interprets TTA as an instance of online parameter evolution, offering a systematic method to understand the process.
46
29
43
2.2 The SDE Approximation
SGD lies the heart of modern deep learning, and understanding its dynamics is fundamental to advancing theoretical and practical training aspects. A growing body of research has leveraged SDEs approximate SGD from continuous-time perspective, providing deeper insights into
learning behavior (3 35, 40 I.. A key development is the use of stochastic modified equations to approximate discrete-time SGD updates with continuous- time SDEs [33]. This formulation captures both the deterministic gradient flow and stochastic fluctuations arising from mini-batch sampling, yielding a more complete description of the learning process. Later studies this approach to various optimizers and used it to analyze the link between learning dynamics and
(35) have generalized
generalization. This work builds upon this perspective and proposes a probabilistic framework that captures parameter dynamics from discrete- time observations in online adaptation.
2

---

3 Parameter Dynamics and Log- Vog-Variance Portraits
In this section, we present a theorem establishing a statistically robust measurement, the log- variance portrait, which tracks the evolution of the parameter distribution. Building on this, Section reveals empirical patterns that characterize stable adaptation, and Section 5 applies these findings develop our adaptation method.
3.1 Online Machine Learning Problem
We consider DNN model f X' - y' parameterized by IRd which maps inputs X' to labels E y. The model defines a conditional distribution p(y æ w), used for inference. Let {< Yn) Do a source model f(: kig) During online adaptation, the model encounters a stream of samples 2n drawn from a sequence target distributions Da * Do at discrete time steps & (1,2,3, K}. The adaptation objective is to minimize the expected risk:
1: No denote samples drawn from source distribution Do. used to pre-train
G(ü, t&) Es, [ ( (((xx1)))1
(1)
where 2i = (a yk) in SL settings and 2 = (xx) in USL settings. The loss function E(-) corresponds to cross-entropy in SL, and to entropy-based objectives in USL. At each time step, the model parameters are updated to minimize the risk:
i (1x)) arg min G(wx,t))
w
The adaptation process implemented using an SGD-based optimizer, producing a time series of parameters (ib((1), ui(l))..... (ix)):)- However, this trajectory is vulnerable to degradation in non-ii.d. environments or USL settings, where biased or noisy gradients can destabilize learning. Our goal is to characterize and improve this trajectory by building probabilistic framework that captures the underlying stochasticity the parameter evolution.
3.2 Probabilistic Framework for Parameter Dynamics
To model parameter evolution during adaptation, we approximate the discrete SGD updates using continuous-time SDE following Li al. [33]. When the learning rate nj is sufficiently small, the parameter updates can be approximated by:
dw((( - ((w, ()di + / (),W),
(3)
where dW, denotes standard Brownian motion, g(w,() VG(w, 1) denotes the gradient of the risk, and E(w, () is the empirical covariance of the gradients. Specifically, we define:
E(w, t) = 1/t Σ (gw as the mean gradient. Following [33), we treat the entries of w as independent and approximate the full covariance matrix with scalar multiple of the identity E(w, t) = o, where of = 1 /d Tr(E,) and Tr(.) is the trace. The parameter distribution p(w(t)) under this SDE evolves according to the Fokker-Planck Kolmogorov (FPK) equation:
(ow T) J(w, +)) with ō(w, t) 1/t Σt= gww, T)
Öp(w(t)) Øt
@p(w(()), (g(w, 1)), Owt
d d
I pp(w(t))
17 [[(w, t)l,,,
2
Owtaw!
1=1
where [:lx and Fly denote vector and matrix components, respectively. The following theorem provides a tractable discrete-time approximation of the transition distribution implied by Eq. B
Theorem 1 (Discretization of the SDE approximation). Let (tx-1,th) tx) be sufficiently small discrete- time interval, and assume that both the gradient g(w, tx) and the variance of remain approximately constant over this interval, denoted g& and of. Then, the transition distribution of the SDE can be approximated by:
Eyj--1); where Al = Lu - -k--1 = n,, and the mean and covariance are given by:
p(ur((x) (w((t=1)) z
(5)
P un((x-1) At, Σ 0f
(6)
with of Tr((gn (gx gx) J. 1. 5k 1/k E 9r-
3

---

-17
Dacorso Time Stop
Deruty
Decmie Tias Stop
Demity
(3) Most Stable: SL, i.dd.
(b) SL, non-i.id.
-200
-205
DiACTTEN Stop
Diaccty
Tiirtees K0mg
(c) USL, i.i..
(d) Most Unstable: USL, non-i.i.d
Loo- variance dynamics under SL and USL (TENT) settings in ii.. and non-ii.d. environ- ments. The blue line shows temporal evolution of log-variance Orange boxes denote the empirical distribution of log-variance The green dash-dot line represents the fitted IG distribution, with red and black dashed lines marking the IG mean and mode, respectively. The degradation in IG alignment reflects increasing instability across scenarios.
Figure 1:
The proof is provided in Appendix[A.]] This result allows us to use discrete-time gradient observations Is approximate the evolution of the parameter distribution, which is originally defined in continuous time. Importantly, it reveals the variance term of At, often ignored in traditional SGD analysis. This variance shows local uncertainty and variability in the direction of the gradient. It is an important signal for assessing adaptation stability. While this variance term provides valuable insight, the raw variance of has extremely heavy tails and is difficult to analyze directly, even in a stable environment (See Appendix A.3). To address this, instead focus its logarithm: U22 log(of At?). The log- variance portrait refers to the empirical distribution obtained by collecting values of 2% at each step during adaptation. This portrait illustrates the evolving distribution of local logarithmic variances in the gradients over time, allowing us to see fluctuations in gradient variability as adaptation progresses. By summarizing these local variances, the log-variance portrait offers compact and interpretable way to visualize and quantify how parameter dynamics change throughout adaptation.
4 Relationship between Log-variance Portrait and Performance
We investigate how parameter dynamics evolve during adaptation by analyzing the behavior of the log-variance portrait across different online adaptation settings. Our goal is to determine whether the aligmment between this distribution and an IG distribution is predictive of model performance, particularly in non-i.i.d. environments. Through extensive empirical analysis, we show that the goodness-off of--itt (GoF) between the log- larvvariance distribution and an IG distribution is reliable indicator of adaptation success. In particular, maintaining high IG-GoF is essential for preventing performance degradation and enabling stable adaptation.
Analysis Setup. We evaluate model dynamics under both i...d. and non-i.i.d. environments, in both SL and USL settings. For USL, we adopt TENT [61] as representative baseline. Experiments are conducted on ImageNet-C (21). which includes 15 corruption-base domains grouped into four broad categories. In the i.i.d. environment, domain samples are shuffled and presented randomly; in the non-i.i.d. environment, domains are presented sequentially in fixed order. To quantify IG-GoF, we fit IG distribution to the log- variance values using maximum likelihood estimation and assess fit using the Kolmogorov-Smme Smirnov test. We report the resulting p-value as the alignment score. The performance improvement measured using the relative average error rate (RAER), defined as: RAER = (Esouce etarget) (angee)//somme /ssonce * 100, where evaunee is the AER of the source model and Guaget is the AER after adaptation. A negative RAER indicates performance degradation. Additional experimental details are provided in Section 6.1

---

0.
06
-155
-160
-16.5
-125
-16
Nackaoor
Nustoncc
(a) Noise (SL)
(b) Blur (SL)
(c) Weather (SL)
(d) Digital (SL)
100
00
000
-27
-20
260 24.3 -263 255 -300
-25
-2
(e) Noise (USL)
(1) Blur (USL)
(g) Weather (USL.)
(h) Digital (USL.
Log-variance portraits for each domain in a non-i.id. environment. The top row shows results under SL, and the bottom row under USL using TENT. The adaptation sequence follows four target domain (i.e., Noise, Blur, Weather, and Digital) with each domain visualized separately.
Figure 3:
Portraits Across All Domains. We first investigate how the log-variance portrait evolves when varying one factor at a time from a stable baseline: SL under i.idd conditions. Figure[[ (a) shows that in this stable setting, the log-variance evolves smoothly over time and aligns closely with the IG distribution. This behavior generalizes across different model architectures and datasets (Appendix [.I and (.6)). When switching to non-i..dd environment (Figure (b)), the portrait partially deviates from the IG shape, indicating destabilization. A similar effect is seen when moving from SL to USL (Figure (c)). When both non-i.i.d. input and unsupervised adaptation are combined, as in realistic TTA scenarios, Figure [ (d) shows a significant breakdown in IG alignment, often yielding multimodal distributions. Takeaway 1.I: Both non-i.i.d input streams and unsupervised objectives introduce instability into the adaptation process. IG alignment in the log-variance portrait generally preserved only under stable learning conditions.
Performance Across All Domains. We next exam- ine how IG-GoF relates to adaptation performance. In Figure 2 lower p-values (weaker IG-GoF) consistently correspond to lower RAER (worse performance), while stronger alignment (higher p-values) correlates with perfor- mance improvements. Notably, the most severe degrada- tion occurs under the combined non-i.i.d. and USL setting, highlighting the compounding effect of sequential domain shifts and lack of supervision. Takeaway 1.2: High IG- GoF is a strong predictor of adaptation success. The log- variance portrait thus provides statistically grounded diagnostic signal for evaluating adaptation quality. These findings naturally lead to the next question: Can degrada- tion in IG-GoF be traced to specific domains encountered during online adaptation?
(.L.))
SL (Non 11.d)
-20
Relative Average Error Sate 0%
0.08-
USL (LL.L))
USL (Non-11dd))
0.05
-10
0.02
Figure 2: RAERS and corresponding p- p-values evaluated across all target do- mains for each adaptation setting.
Portraits and Performance Across Individual Domains. To localize adaptation instability, we examine IG-GoF across individual domain groups in ImageNet-C: Noise, Blur, Weather, and Digital. The order reflects the temporal sequence in which the model encounters each group during online adaptation. Figure 3 visualizes the log-variance portraits for each domain under SL (top row) and USL (bottom row) settings in the non-i.i.d. environment. In the SL setting, the portrait initially aligns well with the IG distribution in the Noise domain but progressively deteriorates, with pronounced collapse in the Weather domain, indicating growing instability in parameter dynamics. Under USL, a similar trend is observed, with the Weather domain again showing the most severe deviation. Notably, the portrait becomes multimodal in this case, diverging from the unimodal structure characteristic of an IG distribution.
5

---

0,30
-18
RDED RAER
0.00
103
0.2
0.2
0.00
0.0
0.0
0.0
0.50
Loy Vactance
Log Vastance
Log Varaa500
Method
(a) TENT
(b) DeYO
(c) ROID
(d) RAER and p-value
Figure 5: Log- Log-variance portraits (a-c) and corresponding RAER and p-value statistics (d) for different TTA methods under the USL setting in a non-i.i.d. environment. Subfigures (a-c) show the empirical log-varian log-variance distributions (orange boxes) and their alignment with the fitted IG distribution (green dash-dot line) for TENT, Do,O, and ROID, respectively. Subfigure (d) summarizes the RAER and IG-GoF (p-values) for each method.
Figure 4 quantitatively summarizes this trend, show- ing the RAER and p-values per domain. We observe that domains with p-values above approximately 0.01 exhibit positive RAER, indicating successful adapta- tion, whereas domains below this threshold experi- ence performance degradation (RAER < O), indicat- ing performance degradation. Takeaway 2: Maintain- ing sufficiently high IG-GoF is essential for avoiding performance degradation. Deviation from the IG dis- tribution is tightly coupled with adaptation failure, reinforcing the importance of preserving statistical structure in parameter dynamics throughout the adap- tation process.
Sl. p-telom
USI peniuw
SI. BARI
USI, BABR
0.10
20
Holetive Average aeragg ETTOF Hate (%)
0.08
0.06
22
-30
0.00
Noise
Blur
Weather (-)
Digital
Adaplation
Figure 4: RAERs and corresponding p-values for each domain under a non-i.id. envi- ronment. Domains with p-values below the threshold (==0.11) tend to experience perfor- mance degradation.
Portraits and Performance of Methods. Recent sample-filering TTA methods have demonstrated strong empirical success in miti- gating the effects of non-i.i.d environment. To better understand this success, we examine how these meth- ods influence the IG-GoF the log-variance distribution. Figure (a c) compares the Vorvvariance portraits of the baseline method TENT with two modern sumple-filterng approaches: Ye [29] and ROID 4 Unlike TENT, which shows apparent deviation from the IG form, both DeYO and ROID yield distributions that are more closely aligned with the IG shape, indicating greater statistical stability. Fig reports RAERS and the corresponding p- values for each method. Both De and ROID achieve substantially higher p-values and improved RAERs compared to TENT, confirming that stronger IG-GoF is associated with better adaptation performance, Takeaway 3: Sample-fittering methods implicitly encourage IG alignment in parameter dynamics. This alignment underlie their robustness in challenging non-i.i.d. environments. Nevertheless, ROID, despite its relatively strong IG-Gof still exhibits minor deviations from the ideal IG distribution. This gap suggests that adaptation performance could be further enhanced by explicitly regulating parameter dynamics to maintain IG alignment throughout the learning process, rather than relying on implicit regularization effects.
5 Structured Inverse-Gamma Model Alignment
We introduce SIGMA, a principled algorithm designed to correct degraded parameter trajectories under USL or no.-i.i.d. conditions, thereby enabling stable and high-performing adaptation. Building on our empirical insights from the previous section, SIGMA explicitly regulates the evolution of model parameters to preserve statistical alignment with the IG distribution, a property closely associated with successful adaptation. In the remainder of this section, we describe the algorithm's 's components in detail and analyze how it dynamically modulates parameter updates through time-varying adjustment of the update interval.

---

5.1 Algorithm
As established in the previous section, high GoF between the log- varvannneee portrait and the IG distribution is a key indicator of stable adaptation. Based on this observation, we hypothesize that adaptation can be improved by (a) identifying an appropriate IG distribution in real time, and (b) actively regulating the parameter dynamics to preserve alignment with it throughout the learning process. SIGMA implements this principle via three core steps: (1) Estimate Step: online estimation of an IG distribution that best fits the observed log-variance dynamics; (2) Align Step: computation of correction coefficients to align the current log- variance with the estimated IG distribution; (3) Conjugate Step: modification of the parameter update based on the aligned distribution. These steps collectively produce dynamic rescaling of the effective update interval At at each time step, thereby inducing statistically calibrated trajectory for parameter adaptation. We begin by reparameterizing the log- variance 2gk at each time step using two correction coefficients: a location shift c₁: S 0 and a scale factor by= >> 1. The corrected log- vagrviaie is defined as:
Úy- (Uk- c2) c + +2 log X,
(7)
where alignment strength X quantifies how closely the updated distribution should follow the real-time reference distribution, which is assumed to follow the IG form.
Estimate Step. We aim to identify an distribution (G; 04-1,, 1) that best explains the history of calibrated log-variances U1 This strategy is framed as a KL divergence minimization:
q(0102.A-1:04-) arg min DEL (p( u/ 01.k-1) ((((#1x-1;;a)
(8)
4EIG
We solve this optimization using the Nelder-Mead simplex method [4]. a derivative-free algorithm well-suited for scalar-valued observations.
Align Step. Given the estimated IG distribution, we compute optimal correction coefficients (ef, bt) that align the current log-variance Eg with the target distribution by the simplex method. This optimization formulated as the negative log- liceliocl minimization:
Ggp bg = arg min - log q(Tx |u 0x=-)) (cu.bu)
We impose constraints o S 0 and 1/ > to ensure stable reparameterization, and clip the solution it falls outside the valid region. Applying these coefficients, the corrected log-variance becomes:
Uk = 2 log ao (ArxTr) = 2 log lox Alul,
(10)
Ate
where the adjusted time interval is defined as Ale (A with Th = exp(-ci/2bi) and
Th = (tt The derivation is detailed in Appendix B.2 This yields the modified transition distribution:
p"
-NWW(NN((w)tttt E 1Rja-1).
(11)
where
Pa-k-1 w(b+-1) gk Alz. Enk-11 - A...
(12)
The effective update interval A/ is shaped by two opposing influences: (1) ra > 1, which enlarges A4; to correct location misalignment: and (2) T < 1, which reduces At; in response to high gradient variance. Together, these terms adjust the parameter trajectory to preserve IG conformity and enhance adaptation stability.
We integrate the corrected dynamics into the parameter trajectory until fx. Using the modified transition distribution, we recursively calculate the marginal distribution follows:
Conjugate Step.
p(w((t)) = /
ur ((t = (unn((+))u: Ex)
(13)
where μ4 = wo _ At, and Σg En + E Af with Σg O and initial prior p(w((o))) = - ((w. En). This resulting distribution over parameters is considered conjugate prior for the predictive distribution:
p(ylan) / plyla w(( (())))))((()()))))))))
(14)
To proceed, we apply the plug-in approximation 44 we evaluate the integral at the posterior mean μ4k yielding the final prediction [(ax: (x) used for risk computation in Eq. ( The algorithm then proceeds to the next time step 44+1, iterating the Fstimate-Align-Con Align-Conjuuate cycle. Illustrations of the algorithm and pseudocode are provided in Appendix B
cycle
7

---

(a) SL
(b) USL, TENT
(c) USL, DeYO
(d) USI, ROID
Figure 6: log-variance portraits under non-i.idd environments for SL and USL settings, before (orange) and after (green) applying SIGMA. Green distributions exhibit closer alignment with the IG reference, indicating improved parameter dynamics.
5.2 Effectiveness
We evaluate SIGMA in both SL and USL settings under non-i.i.d. environments. For the USL case, we integrate SIGMA with three representative TTA meth- ods: TENT, YO, and ROID. Figure O illustrates qualitative effect of SIGMA on the log-variance por- traits. In all cases, SIGMA reshapes the distribution to more closely follow the IG structure, validating its mechanism of action. Even robust baselines such as ROID benefit from additional stabilization when augmented with SIGMA. Figure 7 shows the quan- titative improvements in both RAER and IG-GoF (p-values). Across all tested methods and conditions, SIGMA consistently increases the p-value, indicating improved statistical alignment with the IG distribu- tion. This alignment is invariably accompanied by corresponding gain in RAER, demonstrating that better IG-GoF leads directly to improved adaptation performance. These results support our central hypothesis: preserving alignment with the IG distribution is key to preventing collapse and enabling robust online adaptation. SIGMA functions as a distribution-awan mechanism that enhances both statistical regularity and practical performance across diverse adaptation settings.
poslec SIGMA
-30
Helative Ave Average ETTOR Rate 0%)
p p-lleo N SRGMA
[iguurtiilsssratess the
0.4
BATR w/o SIGMA
HAAEE M SIGIA
-25
0.3
0.1
0.0
SL
USL (TENT)
(D4Y0)
USE (ROOD)
Figure 7: RAERS and corresponding p-values, before (slash-pattern bars) and after (vertical line-pattern bars) applying SIGMA.
6 Comparison with State-of-the-art TTA Methods
In the previous sections, we demonstrated the effectiveness of SIGMA across a range of settings. To further validate our approach, we follow the standardized TTA benchmark protocol 42 under various scenarios. All reported results are averaged over four random seeds, and both AERs and corresponding standard deviations are presented ensure statistical reliability and reproducibility. All experiments were conducted using single NVIDIA GeForce RTX 4090 GPU.
6.1 Experimental Details
Datasets We evaluated SIGMA across both multi-domain and single-domain datasets. For multi- domain adaptation, we used ImageNet-C and D109; for single-domain evaluation, we used Rendition (2 and Sketch 21.. Multi-domain adaptation ImageNet-C is the default dataset; other datasets are used in Appendix C ImageNet-C extends the original ImageNet dataset, consisting of 1,281, 166 training images and 50,000 test images, by applying 15 types of corruption (c.g., Gaussian noise, shot noise, defocus blur, frost, JPEG compression) at five severity levels. Following standard practice we used severity level and treated each corruption type as a distinet domain. The default domain order is Noise, Blur, Weather, and Digital. D109, derived from DomainNet [48). consists of five natural domains (i.e., clipart, infograph, painting, real, and sketch) and contains 109 classes overlapping with ImageNet. This dataset enabled evaluation under real-world domain shifts. To assess adaptation in single-domain settings, we adopted: Rendition, which comprises 30,000 stylized renderings of 200 ImageNet classes curated via Amazon Mechanical Turk; and Sketch, which contains 50,000 black-and-whih sketches across 1,000 ImageNet classes collected via Google Image Search.
45
29 4
8

---

Table AERs (%) and corresponding standard deviations in Correlated Input on ImageNet- C. The bold number indicates the best result.
Noise
Blur
Weather
Digital
Method
grsssian
shot
impulse
defocas
glass
modion
zoom
swww
Trust
Fog
bright
vontrtel
clastic
pixelate
jpe
AFR
Souree
45.9
00.1
36.3
26.3
50.6
67.6
45.4
51.8
RoTTA
45.9
43.3
43:3
092
71.8
59.4
68.7
3938
42.5
35.9
26.2
49.8
66.5
0022
43.4
51.4200002
SAR
44.1
41.8
43.7
69.7
77.3
57.1
66.8
41.1
41.5
42.3
26.3
50.2
64.1
57.2
42.0
51 13,200 30
TENT
43.3
41.1
43.3
00.4
76.3
573
67.1
4013
44.7
51.1
27.3
47.1
64.3
58.6
45.2
X19=070
SIGMA
63.8
42.9
43.0
67.9
73.3
56.2
64.8
39.8
56.8
37.8
26.1
43.5
68.1
49.6
38.5
50.15049
$6.7
30.9
45.4
$0.2=0000
SIGMAA
03.3
37.5
37.1
32.3
23.9
40.8
53.8
35.9
64.820004
DeYO
43.6
66.3
70.3
54.7
62.7
38.3
38.7
36.9
24.6
43.1
55.0
50.1
99.2
47 17.1-
SIGMA
43.6
42.3
41.8
65.4
68.
53.5
65.3
39.3
37,9
32.2
24/6
41.2
58.4
46.
36.9
46. 26.5.0:3
ROID
42.8
40.5
59.8
63.0
63.3
49.8
56.6
56.8
16.3
91.4
24.8
59.6
56.6
47.2
36.5
44 3-0
SIGGMA
42.7
40.3
39.6
61.1
57.0
46.9
518
36.1
34.4
30.5
23.1
38,3
49.1
43.1
33.0
41.8:0.14
Scenarios. We considered two representative unsupervised online adaptation scenarios (i.c., contin- ual TTA setup), each simulating different forms of real-world distribution shift. In the Correlated Input setting, domain-wise data were presented to the model in a fixed sequence, simulating tempo- rally evolving input distributions 6 62 66 In the Correlated Label setting, samples were drawn from localized label distributions generated by Dirichlet distribution with concentration parameter y [17 |6 688. Lower values of y produced stronger label locality and non-stationarity, while 7 = oo reduced to the Correlated Input scenario.
Methods. We compared -GGMA-aagmented methods against range of representative TTA baselines: TENT [61] minimizes entropy loss using the model's 's own predictions to reduce uncertainty during adaptation. RoTTA [66] employs student-teacher architecture with cross-entropy loss and data augmentation, training a student model to adapt while maintaining alignment with a stable teacher. SAR (46) integrates sharpness-aware minimization avoid sharp local optima and resets the model to the source checkpoint when loss exceeds a predefined threshold. EATA 45) extends TENT by filtering out high-entropy (i.e.. low-confidence) samples based on a fixed threshold prevent collapse. Yo [29] identifies reliable samples using entropy and pseudo-label consistency under object-destructive transformations. ROID [K] regularizes entropy account for label distribution diversity and filters low-confidence samples during training. To ensure a fair comparison, we disabled ROID's 's optional prior correction module in all experiments, as orthogonal and applicable to other methods.
we used the base version of Vision Transformer (ViT) [2] with the self-supervised D2V model as our default backbone. We also evaluated SIGMA using Swin Transformer (Swin) [37 to assess architectural generalization. All source models were initialized using publicly available weights pre-trained on ImageNet to ensure fair
Implementation Details. Following prior works 45
29 43
comparison and reproducibility. Consistent with previous studies [34 45 46 43
311. we restricted
training to normalization layers, either batch normalization [24] or layer normalization 2 depending on the model architecture. We adopted official implementations and hyperparameters from each method's 's original paper and the standardized TTA benchmark suite 42). When method-specific settings were unavailable for dataset or model, we defaulted the best-performing configuration reported for ROID. We used SGD with a momentum of 0.9 with the source- parammeer averaging 46 43 31 and batch size of 64 as the base optimizer. Learning rates were set to 1.0 x 10 for D2V, 2.5 x 10-4 for VIT and Swin, and 1.0 x 10 for SAR (using the SAM optimizer [3] across both VIT and Swin models. The SIGMA alignment strength À was fixed per method and held constant across experiments unless otherwise noted: x 5.0 x 10 for TENT, 7.5 x 10-5 for EATA and ROID, and 5.0 x 10-5 for DeYO. We implemented the Kolmogorov-Smimat scipy.stata stats. .etaats..ssstt...eeteee.e....... kstest and the Nelder-Mead simplex method via the scipy. bfats.fft.
test via the
6.2 Results
Correlated Input. Table I summarizes the results on the ImageNet-C dataset under the Correlated Input setting, comparing baseline TTA methods with and without SIGMA. Among the baselines, sample-fittering approaches such as EATA, DeYO, and ROID outperformed other strategies, including mudenttttechhe frameworks (i.e., RoTTA (66]) and sharpness-awar minimization (i.e., SAR [46]). ROID achieved the highest baseline accuracy. When we applied SIGMA, all methods showed further improvements. Specifically, TENT, EATA, Ye, and ROID improved by 1.8%, 5.4%, 0.6%, and 2.5%, respectively. These results demonstrate SIGMA's ability to enhance stability by regulating parameter dynamics via IG alignment.

---

Table 2: AERS (%) and standard deviations in Correlated Label (y 0.1) on ImageNet-C. The bold number indicates the best result.
Nosse
Blur
Weather
Digital
Method
grsssian
shot
impulae
defocas
glass
modion
zoom
soww
Trust
Fog
bright
vontrted
clastic
piselate
jpeg
AIR
Souree
43.9
00.1
36.3
50.6
67.6
RoTTA
25.5
41.2
40.8
68.4
71.1
56.3
64.4
39.1
38.3
38.6
28.3
03.4
67.5
67.4
49.4
520000000
SAR
43.9
41.7
40.9
68.4
71.8
55.0
63.4
50.1
79.1
38.8
25.3
44.8
58.0
49.9
59.3
48.0+0 10
TENT
44.3
41.
433
713
78.3
683
42.4
52.0
36.3
502
64.7
68.7
61.2
44.7
354:3138
SIGMA
43.4
41.8
41.6
66.9
75.4
56.0
68.5
41.6
60.9
33.3
25.4
43.0
69.3
50.1
39.6
$0.5.0.16
32.0
57.1
43.3=0000
SIGMA
35.3
30.5
22-7
38.1
45.3
40.90006
DeYO
41.3
52.3
40.3
40.8
26.1
64.3
66.4
42.9
49.0=2.3
SIGMA
41.5
39.7
39.8
60.7
68.9
49.9
60.4
37.1
35.5
30.9
23.6
38.2
51.4
4044
34.8
44.9.0.20
ROID
40.5
10.4
59.3
54.8
55.4
46.4
53.1
35.5
34.7
30.0
23.7
36.
48.0
41
34.9
40 20.00 10
SIGMA
40.2
38.4
38.4
51.6
49.5
41.3
46.9
33.1
32.8
28.8
22.0
35.5
39.7
35.5
30,9
37.720.01
Table 3: AERs (%) and standard deviations in Correlated Label (7 0.0) on ImageNet-C. The bold
number indicates the best result.
Weather
Digital
Method
ganssian
impulse
defocus
glass
mtotion
z00m
saoww:
frost
tog
bright
costrost
clastic
pixelate
jpeg
Source
43,9
43.3
43.4
69.7
78.3
59.6
69.1
40.1
44.3
36.3
26.5
50.6
67.6
6016
43.4
51.8
KOTTA
23.8
6713
74.5
501.
401.
19.3
40.2
74.3
72.4
72.8
51.3
54.620044
SAR
67.6
38.2
67.5
66.0
57.9
50.5e1388
TENT
46.1
43.7
44.0
71.1
79.2
51.6
69.8
43.2
53.1
55.9
30.8
48.7
69.4
09.1
58.9
56.220996
ASIGMA
43.4
41.8
41.7
67.5
75.3
56.3
67.2
401
58.6
33.9
25.4
43.1
66.7
50.1
39.8
50 1=000
EATA
43.5
42.4
42.4
65.9
71.9
$5.9
64.0
40.5
46.5
40.9
25.6
43.1
63.4
51.4
39.9
49 1:0.34
ISIGMA
42.9
42.4
42.8
70.1
71.2
57.5
64.9
40.5
41.7
36.8
20.4
58.7
82.6
47.3
38.6
28990017
DeYO
41.5
38.9
38.9
61.7
61.3
51.8
42.2
39.7
26.5
56.4
57.1
41.3
41.4
47.9.0.57
SIGMA
41.7
61.2
31.2
25.2-0.10
35.9
42.0.0.11
SIGGA
40.8
55.6
34.5
29.9
37.3
43.4
Correlated Label. We then evaluated SIGMA under more challenging scenarios involving shifts in label distribution. Table Table present the results for 0.1 and 7 0.0, which induced increasingly localized label shifts. These settings exacerbated the risk of performance degradation, particularly for methods such TENT. Despite this, SIGMA consistently prevented degradation. Despite this, SIGMA consistently prevented degradation. TENT's performance improved by 6. % and 4.9% under the y settings, effectively reversing collapse. ROID also benefited from improvements of 2.2% and 3.2%, reinforcing the method's broad applicability.
Additional Results. We extended our experiments to test generalizability for diverse model architectures, realistic scenarios, and datasets. The results in Appendix C showed that SIGMA consistently improved performance across various experimental settings. These results supported our central claim: enforeing IG alignment in parameter dynamics prevented collapse and led to consistent performance improvements under online adaptation settings.
7 Conclusion
In this work, we proposed probabilistic framework to model the dynamics of model parameters during online adaptation using an SDE formulation. By discretizing the SDE, we derived a transition distribution that captured the time-evolving behavior of parameters throughout the adaptation process. Our empirical analysis revealed that the log-variance of this transition distribution served as a key indicator of adaptation stability. We found that successful adaptation consistently coincided with strong alignment between the log- vog-variance portrait and the IG distribution. When this alignment broke down, models experienced performance degradation or collapse. Motivated by these findings, we introduced SIGMA, an algorithm that dynamically estimated the target IG distribution and regularized parameter updates preserve alignment. SIGMA achieved this by adaptively rescaling the update interval based on variance dynamics. When applied to state-of-the-ee TTA methods, SIGMA consistently improved performance across architectures, datasets, and realistic scenarios.
This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea
Acknowledgement
References
[1] Ansari, A. F., Heng, A., Lim, A., and Soh, H. Neural continuous- discreee state space models for eregularly-samplee time series. arXiv preprint arXiv:2301.11300 2023.
10

---

[2] Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06453 2016.
[3] Baevski, A., Zhou, Y.. Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self- supervised learning of speech representations. In Advances in Neural Information Processing Systems, pp. 12449-12460, 2020.
[4] Baevski, A., Hsu, W.-N., Xu, Q., Babu, A., Gu, J., and Auli, M. data2vee: A general framework for super-sdee learning in speech, vision and language. International Conference on Machine Learning, pp. 1912, 2022.
[5] Bottou, L. Online algorithms and stochastic approximations. Online learning in neural networks, 1998.
[6] Boudiaf, M., Mueller, R., Ben Ayed, I., and Bertinetto, L. Parameter-free online test-time adaptation. /EEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2022.
[7] Brahma, D. and Rai, P. probabilistic framework for lifelong test-time adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 3582-3591, 2023.
[8] Chaudhari, P. and Soatto, S. Stochastic gradient descent performs variational inference, con- verges to limit cycles for deep networks. In 2018 Information Theory and Applications Workshop (ITA), pp. 1-10. IEEE, 2018.
[9] Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, K., Torr, H., and Ranzato, M. On tiny episodic memories in continual learning. arXiv preprint 1902. 2019.
[10] Chen, D., Wang, D., Darrell, T., and Ebrahimi, S. Contrastive test-time adaptation. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 295-305, 2022.
[11] Döbler, M., Marsden, R. A., and Yang, B. Robust mean teacher for continual and gradual test-time adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7704-7714 2023.
[12] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint (rXiv:2010.11929 2020.
[13] Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv20010.01412, 2020.
[14] Gao, F. and Han, L. Implementing the nelder-mead simplex algorithm with adaptive parameters. Computational Optimization and Applications, 51(1):259-27 2012.
[15] Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D. P., and Wilson, A. G. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in Neural Information Processing Systems, 31, 2018.
[16) Ghunaim, Y., Bibi, A., Alhamoud, K., Alfarra, M., Al Kader Hammoud, H. A., Prabhu, A., Torr, H., and Ghanem, Real-time evaluation in online continual learning: new hope. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11888-11899 1197, 2023.
[17] Gong, T., Jeong, J., Kim, T., Kim, Y., et al. Note: Robust continual test-time adaptation against temporal correlation. Neural Information Processing Systems, 2022.
[18] Goodfellow, I., Bengio, Y., and Courville, A. Deep learning, volume 1. MIT Press, 2016.
[19] Goodfellow, I. J.. Mirza, M., Xiao, D., Courville, A., and Bengio, Y. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv: 122.62211, 2013.

---

[20] Gurbuzbalaban, M., Simsekli, U., and Zhu, L. The heavy-tail phenomenon in sgd. In Interna- tional Conference on Machine Learning, pp. 2964-3975, PMLR, 2021.
[21] Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corrup- tions and perturbations. International Conference on Learning Representations, 2019.
[22] Hendrycks, D., Basart, S., Mu, N., Kadavath, S., et al. The many faces of robustness: critical analysis of -if-o---iitributionn generalization. International Conference on Computer Vision, 2021.
[23] Hernandez, F., Nguyen, V., Ghannay, S., Tomashenko, N., and Esteve, Ted-lium 3: Twice much data and corpus repartition for experiments speaker adaptation. In Speech and Computer: 20th International Conference, SPECOM 2018, Leipzig, Germany, September 18-22, 2018, Proceedings 20, pp. 198-208. Springer, 2018.
[24] loffe, S. and Szegedy, C. Batch normalization:
Accelerating deep network training by reducing internal covariate shift. International Conference on Machine Learning, pp. 448-456, 2015.
[25] Izmailov, P., Podoprikhin, D., Garipov, T., Vetrow, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv: (80300500, 2018.
[26] Khan, M. E. Deep learning with bayesian principles. Tutorial on Advances in Neural Information Processing Systems, 2019.
[27] Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., al. Wilds: A benchmark of in-the-wild distribution shifts. International Conference on Machine Learning, pp. 5637-5664, 2021.
[28] Kushner, H. Approximations to optimal nonlinear filters. IEEE Transactions Automatic Control, 12(5):546-556, 1967.
[29] Lee, J., Jung, D., Lee, S., Park, J., Shin, J., Hwang, U., and Yoon, S. Entropy is not enough for test-time adaptation: From the perspective of disentangled factors. arXiv preprint axXiv:2403.07366 2024.
[30] Lee, J. Bayesian weight enhancement with steady-state adaptation for test-time adaptation in dynamic environments. In Forty-second International Conference on Machine Learning. 2025.
[31] Lee, J.-H. and Chang, J.-H. Continual momentum filtering on parameter space for online test-time adaptation. In The Twelfth International Conference Learning Representations, 2024.
[32] Lee, J.-H. and Chang, J.-H. Stationary latent weight inference for unreliable observations from online test-time adaptation. Forty-first International Conference on Machine Learning. 2024.
[33] Li, Q.. Tai, C., and Weinan, E. Stochastic modified equations and dynamics of stochastic gradient algorithms i: Mathematical foundations. Journal of Machine Learning Research, 20 (40):1-47, 2019.
[34] Li, Y., Wang. N., Shi, J., Hou, X., and Liu, J. Adaptive batch normalization for practical domain adaptation. Pattern Recognition, 80:109-117, 2018.
[35] Li, Z., Malladi, S., and Arora, On the validity of modeling sgd with stochastic differential equations (sdes). Advances in Neural Information Processing Systems, 34:12712- 2021.
[36] Lin. G.-T., Li, S.-W., and yi Lee, H. Listen, adapt, better wer: Source-free single-utterance test-time adaptation for automatic speech recognition. In Proc. Interspeech, 2022.
[37] Liu, Z.,, Lin, Y., Cao, Y., Hu, H., et al. Swin transformer: Hierarchical vision transformer using shifted windows. JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10012-10022, 2021.
[38] Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. P., and Wilson, A. G. A simple baseline for bayesian uncertainty in deep learning. Advances neural information processing systems, 32, 2019.
12

---

[39] Madry, A., Makelov, A.. Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0080 06083, 2017.
[40] Malladi, S., Lyu, K., Panigrahi, A., and Arora, S. On the sdes and scaling rules for adaptive gradient algorithms. Advances in Neural Information Processing Systems, 35:7697-7711, 2022.
[41] Mandt, S., Hoffman, M. D., and Blei, D. M. Stochastic gradient descent approximate bayesian inference. Journal of Machine Learning Research, 18(134):1-35, 2017.
[42] Marsden, R. A. and Döbler, M. test-time- admeaadaptation. http://gghhbbbccommmnnniccccchhs adaptation, 2022.
[43] Marsden, R. A., Döbler, M., and Yang, B. Universal test-time adaptation through weight ensembling, diversity weighting, and prior correction. arXiv preprint arXivv2306.006533 2023.
[44] Murphy, K. P. Probabilistic machine learning: Advanced topics. MIT press, 2023.
[45] Niu, S., Zhang, Y., Chen, Y., et al. Efficient test-time model adaptation without forgetting. International Conference Machine Learning, pp. 16888-16905, 2022.
[46] Niu, S., Wu, Zhang. Y., Wen, Z., et al. Towards stable test-time adaptation in dynamic wild world. ArXiv, 124/20002.14400, 2023.
[47] Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. Librispeech: an asr corpus based on public domain audio books. In IEEE international conference on acoustics, speech and signal processing, pp. 5206-5210. IEEE, 2015.
[48] Peng, X., Bai, Q., Xia, X., Huang, Z. et al. Moment matching for multi-source domain adaptation. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1406- 1415, 2019.
[49] Polson, N. G. and Sokolov, V. Deep learning: A bayesian perspective. Bayesian Anal, 12(4): 1275-130% 1304, 2017.
[50] Prabhu, A., Al Kader Hammoud, H. A., Dokania, K., Torr, P. H., Lim, S.-N., Ghanem, B., and Bibi, A. Computationally budgeted continual learning: What does matter? In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 3698-3707, 2023.
[51] Press, O., Schneider, S., Kümmerer, M., and Bethge, M. Rdumb: simple approach that questions our progress in continual test-time adaptation. Advances in Neural Information Processing Systems, 36, 2024.
[52] Rame, A., Kirchmeyer, M., Rahier, T., Rakotomamonjy, A., et al. Diverse weight averaging for outtof-distrribttio generalization. Advances in Neural Information Processing Systems, 35: 10821-10836, 2022.
[53) Särkkä, S. and Solin, A. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
[54] Särkkä, S. and Svensson, L.. Bayesian Biytsian,illeriig and smoothing, volume 17. Cambridge university press, 2023.
[55] Schirmer, M., Zhang, D., and Nalisnick, E. Test-time adaptation with state-space models. In ICML 2024 Workshop on Structured Probabilistic Inference {{&} Generative Modeling, 2024.
[56] Schneider, S., Baevski, A., Collobert, R., and Auli, M. wav2vec: Unsupervised pre-training for speech recognition. In Proc. Interspeech, 2019.
[57] Simşekli, U.. Gürbüizaalban,, M., Nguyen, T. H., Richard, G., and Sagun, L. On the heavy-tailed theory of stochastic gradient descent for deep neural networks, arXiv preprint arXiv: 19/2000/0 2019.
[58] Simsekli, U. Sagun, I.., and Gurbuzbalaban, M. A tail-index analysis of stochastic gradient noise deep neural networks. In International Conference on Machine Learning, pp. 5827- 5837. PMLR, 2019.
13

---

[59] Simsekli,, U. Sener, O., Deligiannidis, G., and Erdogdu, M. A. Hausdorff dimension, heavy tails, and generalization in neural networks. Advances in Neural Information Processing Systems, 33: 1158-5151, 2020.
[60] Van de Ven, G. M. and Tolias, A. S. Three scenarios for continual learning. arXiv preprint arXiv: 2019.
[61] Wang, D., Shelhamer, E.,, Liu, S., Olshausen, B., and Darrell, T. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint 2006 2020.
[62] Wang, Q., Fink, O., Van Gool, and Dai, D. Continual test-time domain adaptation. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7201-7211, 2022.
[63] Wilson, A. G. The case for bayesian deep learning. arXiv preprint arXiv:2001.10995, 2020.
[64] Wilson, A. G. and Izmailov, P. Bayesian deep learning and probabilistic perspective of generalization. Advances in neural information processing systems, 33:4697-4708,, 2020.
[65] Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Interna- tional Conference on Machine Learning, pp. 23965-23998. PMLR, 2022.
[66] Yuan, L., Xie, B., and Li, S. Robust test-time adaptation in dynamic scenarios. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15922-15932, 2023.
[67] Zhou, A. and Levine, S. Bayesian adaptation for covariate shift. Advances in Neural Information Processing Systems, 34:914 34:914-922 927, 2021.
[68] Zhou, Z., Guo, L.-Z., Jia, L.-H., Zhang, D., and Li, Y.-F. Ods: test-time adaptation in the presence of open-world data shift. International Conference on Machine Learning, pp. 42574- 42588, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We state our claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our study Appendix F.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We clarify our assumptions in Theorem and provide the proofs in Appendix
A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
All the experimental details are provided in Appendix C.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

reproducible, they Answer: [Yes]
Justification: We use publicly available, open-access pre-trained models and datasets for all of our experiments. Additionally, provide pseudocode for the proposed algorithm in Appendix B.I to facilitate reproducibility and implementation.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.cem ec/
public/guidess httss//ooeSummissioooooe.
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 7/nips. //nips.cc/pubb httppss.ccpublli/guuueeee Sttpeee///oooeSeeesseeiiiiii
ssioiiolioiiiiiocyicc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
All the experimental details are provided in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We run the experiment with four seeds and report the standard deviations and average error rates.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We comment on the computer resources in Appendix C and discuss efficiency in Appendix D.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We have read and conform to the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We focus on general and foundational machine learning research.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release pretrained models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: To the best of our knowledge, we adequately cite and mention all used assets in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do provide new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification:
We do not involve crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in our research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
21

---

A Parameter Dynamics and Log- Log-Variance Portraits
In Appendix A.1 we present the proof of Theorem | which characterizes the time-varying parameter distribution during adaptation. Building on this theorem, in Appendix A..3 we compare the dynamics of the raw variance and its logarithmic counterpart (i.e., log-variance). Our empirical observations show that the log-variance offers more stationary and statistically tractable distribution, enabling practical computation of distributional alignment during adaptation.
A.1 Proof of Theorem
Proof. Following Theorem 5.4 in Särkkä & Solin (53, the evolution of the mean m(l) = [[()(()) and the covariance S(t) Cov[w(t)] for the FPK solution p(w, t) from Eq. (4) is given by:
dm
/ g(w, t))(w, t)dw.
dt
ds
f g(w,, t)(w m) p(w, t)dw
dt
/ (w-m)g T (pw,) f)
/ [ww,tLL (w, t)p(w, t)dw,
where L(w,t)= = Vl Assuming a Gaussian approximation to the parameter distribution 28 write:
p(w, t) = (m)
We linearize g(w,, t) and L(w, t) around mtt) via first-order Taylor expansion:
g(w, t) g(m, t) ()w m), L(w, t) =e L(m,t),
where ((m, 1) the Jacobian of g evaluated at m. Substituting these into the FPK evolution yields:
Mean:
dm
= flo(m.t) + t) ((m))
dt
1m,, ((m, / (w W
-
Covariance:
dS
/ [o(m,t) t) G(m, f)w - m))(w m) m) New / (w mllg(m, t) g(m, (w m)] Ndw
dt
+ / L(m,t)L (m, N)wd..
We compute each term separately:
-flomm.) G(m, m)r](w T Ndw tm (w m) New
-g(m, / (w (w-m))(w
m) m) Ndw
--Gmmtt)
(a)
22

---

- -f(v ( m0)lo(mm)) + 9(m, t)( m)rl Wdwl"Ndww = /(w (w-m)))e m))dw g(m.t)
(w m)(w m) TTN New g (m, t)
8 --S -mmggmmm)))
(b)
J t(m (m,,)))(((,t)at t) = L(m, t) (m,
=
(c)
Summing Eqs. (a-c), we obtain the covariance dynamics:
dS dt = Sg (m, t) no
1)) of the SDE a Gaussian distribution following Lemma A.9 in Särkkä & Svensson [54] We consider sufficiently small interval (fg-1, 1, ty) with At t4 - - = 17, and assume that both the gradient (m, t) and the variance term of L(m. t) remain approximately constant over this interval, denoted by I and of Under the mastant-gradieen assumption, we also take G (m, t) O. Solving the resulting ordinary differential equations yields the mean and covariance the transition distribution:
Discretization:
The transition distribution (( [w((x- (lx:)/(((ll)))))))))
Pajk-1 m(tx|tx-1)
pt 1+At
= -w0(1=1)
9 di
1
w((x-1) 1)
Ib. Dt
Mxjk-1 = s(tnltx-1)
tk-1t +tx1
no dt
J
I
where m(x=1)) = w0x--1) and S(4=-1) =
A.2 Justification of Small Variance
Our derivation assumes the gradient and variance approximately constant during each discretiza- tion step. This is justified by the short time interval involved in transitioning from continuous to discrete time.
The time interval determined by the learning rate 7,, and we adopt a learning rate in the range of 10-5 to 10-6, which is about 100 times smaller than that used during source model training (typically 10-3 to 10-1). Empirically, as shown in Figure (x-axis), the maximum observed variance around e- 11 and the minimum is approximately e 25 This result demonstrates that variance remains extremely small on continual TTA settings. This behavior is consistent across multiple datasets, shown in Figure g
This consistency can be explained by the fact that TTA assumes well-trained model its starting point (see Section 2.1), resulting in minimal model drift during adaptation. Consequently, the assumption of approximately constant mean and variance naturally satisfied in the TTA setting.
23

---

A.3
Comparison of Variance and Log-variance
1--0
-33
-14
-17
200
-02
550
1,0 Deenlity
200
1200
0.3
Diecnete Stop
Blormee Timn Stop
Danellly
Variance
Log-variance
Figure 8: Variance (left) and log-variance (right) dynamics under SL settings in i.i.d. environments. The variance signal exhibits strong fluctuations and heavy-tailed distribution, complicating statistical modeling. In contrast, the log-vas variance shows more stationary behavior and aligns well with the IG distribution, enabling stable estimation and interpretation.
B
Structured Inverse-Gamma Model Alignment
B.1 Illustration of Algorithm
At
Parameter
Trajectory
w(fa-s)
Retimatt Stop
Align Step
Dist. (ilaesite
SIGMA
Alg x reTe
To (v.00
Comjugaza Stop
Figure Illustration the SIGMA algorithm. The black crosses represent the model param- eters at each discrete time step, and the gray curves depict their corresponding distributions p(w(f1:))) = ((ws(1)))) μk. Σx). In Estimate Step, SIGMA estimates an IG distribution us- ing the log- log-veriance values f1:k-1 derived from the covariance of the transition distribution p(w((z) [w(+==1)) = N((((t))lM)
Esja-1) up to time tx-1- In Align Step, it computes the optimal correction terms ef and bg to align the current log variance with the estimated IG distribution. In Conjugate Step, these corrections adjust the update interval A4e, thereby controlling the parameter dynamics at time step lg.
B.2 Derivation of Algorithm
In this section, we provide the derivation of Eq. (10). Using Eq. 7 and the log-variance, get
2k Ck
log A 2log(orAt) Ck + 2 log A.
be
by
Substituting and simplifying, we get:
log(o:A() c
log
by
Taking the exponential of both sides:
Tk
log(onAt) bk
exp
= Aexp
exp
= >ArATR
2
2b
where exp (log(a> Al)/ba) = (oxA()2-bu) //bbbb/bbb (ox Al) T(( Taking the logarithm of both sides:
oz 2 (n
24

---

B.3 Pseudocode
Algorithm 1 Structured Pveese-Gamma Model Alignment
Require: Initial interval At, Source model f(: wo), Alignment Strength X, Initialization: f40 = wD Eo = O, 90 0, U1.1 = {
for k to K do
gk - ne G(px.tt)
Eq.
9k [gk 9k-1)
Ok - 1/k 9k
of + 1/d [a [x)
0k - log(o
AtR + At
(ct, bg) + (0,1)
if ÚiLA then
Estimate Step:
(00/11 - <- arg mingeeG (xu(p(v(rrs-=))) (=((0xx)110))
Eq. (8)
Align Step:
ÜK (un /bx log A
(cg, bt) - arg min (cx.ba) log 15
D> Eq. (9)
+ exp(-cf/ (2b)(c)/(2b)))
Th + (ox At) b2b1//b
Ale + (Ara Th) At
end if
Conjugate Step:
(px: Σx) &- (Mx=1 9k Atk. Σg + Af I)
> Recursion of Eq.
13
if k then
Uk < (log(0} A12) /bg + 2lo
01sk 01 {
end if
end for
C Extended Experiments
To evaluate the generality and practicality of our method, we conducted a broad set of extended experiments. These include studies across diverse model architectures, runtime efficiency, domain ordering, repeated datasets, various datasets, and the impact of alignment strength on performance.
C.1 Diverse Model Architectures
(a) VIT
(b) Swin
(c) D2V
Figure 10: Log-varianee variance dynamics for each backbone architecture under the stable adaptation setting (i.e.. SL in i..dd environments) ImageNet-C. The blue line represents the evolution of log variance over time. Orange boxes show empirical distributions. The green dash-dot line denotes the fitted IG distribution, with red and black dashed lines representing its mean and mode, respectively.
Justification of IG Alignment. the log-variance under the stable adaptation setting (i.e., SL in i.i.d. environments) using three different backbone architectures: VIT, Swin, and D2V. shown in Figure TO, the temporal evolution of log-va log-variance (blue line) exhibited distinct patterns across models. However, in all cases, the
verify the robustness of IG alignment assumption, we measured
25

---

resulting log-variance portraits (orange boxes) showed strong GoF with the IG distribution (green dash-dot line). This consistency confirms that IG alignment a model model-agnostic property of stable adaptation dynamics.
Table 4: AERS (%) and corresponding standard deviations for VIT in Correlated Input on ImageNet-C. The bold number indicates the best result.
Noise
Blur
Weather
Digiol
ganssian
shot
impuise
defocus
glass
motion
z00m
snow
frost
log
bright
contrest
clastic
pixclate
Jpeg
56.9
2933
57.1
65.8
55.6
44.1
5003
SAR
61.3
55.7
54.4
62.0
61.4
53.8
57.0
53.9
45.1
45.9
29.1
55.1
51.5
49.2
40.3
51.7=0.14
TENT
63.8
61.7
59.9
67.2
71.0
50.3
63.3
55.6
16.5
47.9
28.5
71.8
55.2
54.5
46.5
57.020.04
ISIGMA
63.8
61.7
59.3
63.9
64.2
53.9
57.3
53.9
44.6
40.4
27.4
59.1
52.9
47.6
41.0
52.8=0.077
EATA
61.8
57.1
56.2
61.5
62.5
54.4
57.2
51.8
45.0
42.5
28.0
56.8
51.7
48.4
42.4
51 81...0.10
SIGMA
61.8
55.5
57.6
29.2
43.8
40.8
31.6
26.7
48.0
46.2
36.5
47.420.05
IIGGMA
60.8
57.4
53.6
37.5
47.2
ROID
57.6
31.5
52.3
55.1
52.4
46.5
47.3
45.6
39.5
36.0
26.0
45.0
43.8
36.3
SIGMAA
57.9
31.7
52.1
54.4
50.5
45.3
44.4
44.3
39,4
35.5
26.1
45.3
39.6
37.5
35.2
43.9:0013
Table 5: AERS (%) and corresponding standard deviations for Swin in Correlated Input on ImageNet- C. The bold number indicates the best result.
Noise
Blur
Weatther
Digital
Method
ganssian
defocus
frost
Fog
clastic
jpeg
AEK
Source
71.0
70.0
15.4
72.8
$1.6
63.8
68.2
57.9
50.7
40,2
28.6
6016
72.1
56.6
59.3
64.0
RoLTA
71.0
69.3
73.8
73.2
80.4
52.7
67.2
56.9
18.7
42.9
29.1
59.0
69.2
55.7
59.00
63.40001
SAR
63.5
77.1
65.5
63.3
70.2
71.8
63.0
63.9c1
TENT
78.2
57.0
63.5w0.00
ASIGMA
67.3
62.7
61.6
55.5
51.2
26.7
50.4
EATA
63.0
56.8
57.6
68.4
66.8
54.6
55.7
52.3
46.7
42.1
25.9
48.8
57.1
64.8
49.7
54.0e0.13
ASIOMA
63.0
56.8
57.2
65.9
63.9
$2.9
50.9
48.6
48.4
39.8
24.8
46.5
50.9
58.5
44.3
51 21.000005
DeYO
62.6
36.8
$1.3
72.7
68.9
58.3
62.4
52.8
46.7
76.3
26.3
48.3
58.2
64.4
49.5
57.821.00
ASIGMA
62.6
56.8
57.3
71.8
68.9
56.7
62.2
$1.6
45.0
65.1
25.9
47.6
53.5
60.3
46.9
55 155
36.2
43.9
47.0e0 26
ASIMAA
Benchmark Results. Building on IG alignment principle, we applied SIGMA to both ViT and Swin backbones across multiple TTA methods. Table 4 reports results ViT. SIGMA improved performance by 4.2% with TENT, 4.4% with EATA, 0.8% with DeYO, and 1.1% with ROID. Similar trends were observed in Table 5 Swin, with gains of 2.1% % (TENT), 3.8% (EATA), 2.2% (DeYO), and 0.9% (ROID), respectively. These consistent improvements reinforce the generality of IG alignment principle and its effectiveness across different model architectures.
C.2 Runtime Efficiency
Efficiency is critical factor online adaptation scenarios. To quantify runtime overhead, we use GPU wall time (in milliseconds per sample) as efficiency metric. Figure| II illlustrates the wall time and RAER before and after applying SIGMA for each sample-fitering method un- der the Correlated Input scenario on ImageNet- C. SIGMA introduced minimal additional over- head (e.g.. only 0.04 ms sample for EATA and ROID) while achieving substantial RAER im- provements of 10.4% and 4.8%, respectively. These results demonstrate the high efficiency of SIGMA, which stems from its lightweight de- sign based on scalar variance observations and derivative-free free optimization. Overall, SIGMA maintains a strong balance between computa- tional cost and adaptation effectiveness.
TENT
20
FATA
ROID
13
BATA+ (0.24 ItS. 14.5%) (0.21 13,5%)
Ralative Average
(1.2 9.1%)
BATA
3,3%)
(0.16 TINT
0.18
0.20
0.22
0.24
0.28
CPU Wall Time (ats/aamppa)
Figure 11: RAERs and corresponding GPU wall time for each method w/o and w/ SIGMA on ImageNet-C.
26

---

C.3 Various Domain Ordering
In real-world online adaptation scenarios, the sequence in which domains appear can vary sig- nificantly, and robustness to such variation is essential for practical deployment. To evalu- ate this, tested different permutations of the four broader domain categories in ImageNet-C C: Noise (N), Blur (B), Weather (W), and Digital (D). Specifically, we evaluated four domain or- derings: the original order N B W D, along with three alternative sequences (i.e., B W -D N. W + D N -> B, and D N
54
TENT
ONTO
Averoge Error Rate (%)
-22
50
48
16
44
B - W). Figure 12 presents the AERs for each TTA method with and without SIGMA across these sequences. In all cases, SIGMA consis- tently improved performance, regardless of do- main order. These results underscore SIGMA's robustness to domain sequence variation and confirm its practical utility in dynamic and unpredictable real-world environments.
N-B-WW-D
B-W-D-N
Wi-D-N-BB
D-N-B-W
Order
Figure 12: AERs in different domain orders for each method w/o and w/ SIGMA on ImageNet-C.
C.4 Impact Alignment Strength on Performance
To examine the impact of alignment strength, we focused on ROID and systematically varied the value of the hy- perparameter À by introducing scaled perturbations. We
- Maer STD
Point: (AF8-53 690
Averaaee Brrur Bate
began with base value of À 7.5 10-5 and added an increment ne, where с = 10-8 and n increased multiplicatively by factor of 2. Figure [3] presents the AERS and corresponding standard deviations for each À setting. We observed that increasing alignment strength from n 2 to n 8192 gradually reduced the AER from approximately 41.7% 40.8%, indicating improved performance. However, when reached 16384, the AER sharply increased to 53.6%, demonstrating that excessive alignment strength impaired performance. These results confirmed that moderate increases in alignment strength enhanced adaptation, while overly aggressive regulariza- tion degraded Consequently, SIGMA maintains robust values as long overly aggressive regularization is avoided.
2-
120
512
1034
20018
Figure 13: Effect of additional alignment strength on AER. Green circle markers denote mean AER; red bars indicate stan- dard deviation.
performance across a broad range of
C.5 Repeated Datasets
To assess practical applicability in real-world usage pat- terns, evaluated repeated dataset scenario in which the same sequence of domains was presented to the model multiple times. This setting mimies recurring distribution patterns commonly observed in daily tasks or seasonal environmental cycles. We compared the performance of ROID with and without SIGMA under repeated domain exposures. As shown in Figure 14 ROID maintained sta- ble adaptation performance across repetitions. Notably,
SIGMA
Nourage
2.
8
10
12
14
The Number of Datasot Repetitions.
SIGMA consistently outperformed ROID throughout the entire sequence, keeping the AER below 41.5% across all 15 repetitions. These results suggest that SIGMA not only prevents performance degradation during initial adaptation also sustains robust performance in long-term, cyclical deployments, demonstrating strong potential for real-world online adaptation scenarios.
AERs over repeated Ima The maximum number of repetitions is 15.
Figure 14:
27

---

C.6 Various Datasets
-25
(a) Rendition
(b) Sketch
(C) D109
Figure 15: Log-variann variance dynamics under the stable adaptation setting (i.e., SL i..d. environments) on Rendition, Sketch, and D109. The blue line represents the evolution of log-variance over time. Orange boxes show empirical distributions. The green dash-dot line denotes the fitted IG distribution, with red and black dashed lines representing its mean and mode, respectively.
Justification of IG alignment. To validate the generality of IG alignment assumption, measured the log- varianeee distributions under the stable adaptation setting (i.e., SL i.id. en- vironments) across diverse datasets, including Rendition, Sketch, and D109. These datasets span both single-domain (i.e., Rendition and Sketch) and multi-domain (i.e., D109) scenarios. As shown in Figure [5] the empirical distributions of log- variance (orange boxes) across all three datasets exhibited strong alignment with the IG distribution (green dash-dot line). These findings confirms that IG alignment assumption holds across different domains and types of distribution shift.
Table 6: AERS (%) and correspond- ing standard deviations on Rendi- tion and Sketch.
Method
Rendition
Sketch
Single-Domain Settings. We evaluated SIGMA on the Ren- dition and Sketch datasets by applying it to ROID, which served as a strong baseline. As reported in Table 6 SIGMA sig- nificantly improved AERs compared to ROID, achieving the best results on both datasets. These findings demonstrate that SIGMA remains effective even in specialized single-domain adaptation scenarios, further reinforcing the robustness of IG alignment principle.
Source
46.6
60.4
RoTTA
46.5+0.01
60. 10..03
SAR
45.940.05
60.2+0.07
TENT
46.0+0.03
60.3+0.06
EATA
45.8+0.09
58.640.08
DeYo
42.9+0.07
60.420.62
ROID
41.4+0.08
55.7+0.02
SIGMA
37.9*0.09
51.5+00.2
Table AERs (%) and corresponding standard deviations in Correlated Input on D109.
Table 8: AERs (%) and corresponding standard deviations in Correlated Label on D D109.
Mathod
dipart
infograph
painting
rell
sketch
AFR
Merhod
clipont
infogmah
painting
real
sketch
AFR
Source
48.7
72.9
41.1
20.5
56.7
48.0
Source
12.9
41.2
20.5
56.7
48.0
RoTTA
18.6
72.6
10.7
19.9
53.9
47.220.01
ReTTA
45.7
72.7
10.9
20.2
55.4
47.610.03
SAR
48.3
74.3
42.9
20.3
56.5
48.510.10
SAR
48.4
74.6
43.5
20.3
56.4
48.6 18.61002
62.8:0.20
TENT
EATA
17.9
71.6
39.9
19.7
54.1
16.620.04
EATA
47.8
71.5
39.9
19.8
53.7
46.510.06
DeYO
47.2
74.4
40.9
19.7
51.2
46.710.13
D-YO
47.3
74.4
40.6
19.7
$1.0
46.610.40
ROID
43.4
68.5
37.6
19.3
50.4
43.8:0003
ROID
43.4
68.0
37.7
19.4
50.5
43.8:006
SIGMA
183
44.6
41,450003
SIGMA
41,850066
Multi-Domaie Settings. To evaluate SIGMA under natural multi-domain shifts, we used D109, dataset that features stylistic variation across domains rather than synthetic corruptions. In Table (Correlated Input) and Table 8 (Correlated Labels, y 0.1), SIGMA consistently outperformed ROID, confirming its effectiveness in realistic domain shift conditions. These results show that SIGMA is robust to natural distribution changes and validate the broad applicability of the IG alignment assumption beyond synthetic corruption-based henchmarks.
Table 9: AERS (%) and corresponding standard deviations on CIFAR1O-C, CIFAR100-C and ImageNet-C.
Dataset
Source
CoTTA
PETAL (FIM)
RMT
ROID
SIGMA
CIFARTO-C
43.5
16.5+F0.16
16.03
17.0
16.3+0.17
15.7+0.06
CIFAR100-C
46.4
32.8++0.77
31.3++0.13
30.6+10.11
31.7++.11
29.6 0.07
ImageNet-C
64.0
59,3+1123
58.310.14
52,6+11.00
47.0+10.6
46.1
28

---

Small Multi-Domain Settings. We conducted experiments on CIFARIO-C and CIFARI00-C for ResNet-based models (2.) For PETAL Z7 we adopted the FIM variant. Table 9 presents comparison between student-teacher methods and SIGMA applied on top of the strongest entropy- based baseline, ROID. These results show that ROID outperforms student-teacher approaches on ImageNet-C,, which involves larger number of classes. In contrast, on CIPARI0-C and CIFAR 100- C, student-teacher methods such as PETAL and RMT [ exhibit error rates lower than ROID. Specifically, PETAL achieves better performance than ROID on both CIFAR datasets, and RMT outperforms ROID on CIFAR IO0-C. However, SIGMA consistently improves ROID performance in all datasets, achieving error rates lower than those of all student-teachee methods, including CoTTA, PETAL, and RMT. These results demonstrate that SIGMA not only offers computational efficiency but also delivers superior accuracy compared to student-teacher models.
C.7 Real-world Scenario
We evaluated several TTA methods in real- time speech recognition scenario using the TEDLIUM3 dataset [23] containing streamed TED talk recordings. The validation and test sets included speech from 8 and 11 speakers, each covering different topics. Our experiments used
Table 10: Average word error rates (WER, %) and standard deviation on validation (VALID) and test (TEST) sets in the real-world speech recognition scenario on TEDLIUM3.
Method
VALID
TEST
version of the D2V model pre-trained on LibriSpeech [47] as the backbone. Following the experimental protocol of SUTA (36. an established TTA method for speech recognition, we simulated realistic speaker adap- tation by measuring average word error rate (WER) as new speakers sequentially entered the stream. As shown in Table 10] SUTA+SIGMA consistently outperformed all baselines, achieving the lowest WER on both the validation and test sets. These results demonstrate that SIGMA enhances real-time adaptation by effectively mitigating speaker shift, confirming its practical applicability to real-world online adaptation tasks such as speech recognition.
Source
13.0
12.4
Pseudo Label
1207
0.03
TENT
12.5
0.09
SUTA
12.4
1106
SUTA+SIGMA
12.3 + 0.01
11,4 0.01
D Related Works
D.1 Learning Dynamies
From the perspective of learning dynamics, a growing body of work demonstrates that the noise introduced by SGD exhibits heavy-tailed behavior [58] 59 20 57). These studies establish that the tail behavior of SGD noise is closely related to the flatness of loss minima and, by extension, generalization performance [59 20 This strong theoretical understanding underpins the empirical observations reported in Appendix A.3 and substantiates our approach of treating variance as a random variable rather than fixed value.
In parallel to these findings regarding learning dynamics and noise behavior, recent work has reinterpreted SGD-like updates from Bayesian perspective. For instance, the existing works [41 g model SGD as an SDE where the stationary distribution approximates Gaussian posterior. This work offers a principled explanation for the implicit regularization effects of SGD. Based on this view, Maddox et al. (38] have proposed posterior approximations using the trajectory of SGD itself. This work fits a Gaussian to the empirical mean and covariance of SGD iterates to enable practical Bayesian ensembling. These studies mainly focus on training from scratch or in supervised learning settings.
Distinct from prior work, we decisively extend the Bayesian viewpoint to the demanding setting of sequential domain shifts and unsupervised online adaptation. Our empirical evidence establishes that the log-variance of parameter transitions consistently aligns with the IG distribution under stable conditions. Motivated by this observation, we propose SIGMA, an algorithm that leverages this maintained alignment to deliver both stability and flexibility during online adaptation, setting apart from existing approaches.
29

---

D.2 Bayesian Deep Learning
DNNs are highly flexible, allowing them represent many functions with varying levels of general- ization. Using this implicit capacity is key to improving adaptability to shifts in data distributions. This concept is often applied in Bayesian deep learning, which treats model parameters as samples 64 26]. Usually, the parameter distribution is approximated as a Gaussian centered around the parameter mode [8]. With this assumption, repeated training produces multiple models, which are then aggregated. Model averaging, based on this approach, has
from an underlying distribution [49 63
shown strong robustness to distribution shifts [25]
15
65] 52].
Recent studies have extended the Bayesian perspective to the TTA setting by analyzing changes
For example, Lee & Chang 31 32 adopts continual learning viewpoint, applying Bayesian filtering while explicitly fixing the transition distribution balance information from past and present tasks, Other work (55] designs state-space models that are fundamental to Bayesian filtering and directly learn the transition distribution from the data. More recently, Lee [30] proposed modeling the transition distribution via an SDE, incorporating Bayesian filtering where the posterior is forced to converge toward fixed value for stable adaptation. However, such approaches often overlook the natural dynamics of model parameters during real-world online adaptation. Moreover, this work forced convergence excessively
in parameter distributions during online adaptation 31 32 55 30).
suppresses heavy-tailed behavior, which degrades adaptability [58 59 20 57].
In contrast, our work directly treats the variance of the transition distribution as random variable and explicitly analyzes its temporal evolution during online learning. As result, we uncover key empirical law: the log-variance of parameter transitions aligns well with an IG distribution under stable online conditions. Building on this observation, we introduce a principled algorithm that estimates the IG distribution in real time and modulates the parameter dynamics accordingly. Consequently, by modeling variance as a stochastic quantity, our method avoids oversuppressing heavy-tailed behavior, thereby preserving adaptability. Furthermore, unlike conventional Bayesian filtering approaches, our method introduces latent variables, making it lightweight and memory efficient.
E
Limitation and Future Work
One limitation of our approach is the assumption that the parameter distribution follows a Gaussian form. While this assumption enables analytical tractability, it may limit the expressiveness of the underlying distribution. Nevertheless, our empirical results demonstrate that SIGMA, built upon this approximation, consistently performs well across various realistic scenarios and model architectures. We aim to extend our probabilistic framework for future work to account for more complex, non- Gaussian parameter distributions. We anticipate that such generalizations will be necessary in applications where greater distributional flcxibility is critical. This direction may lead to more general and consequential online adaptation techniques.
30


Reviews:
[{"review_id": "J5A0gMgOTZ", "reviewer": "Reviewer_ePCh", "summary": "The authors empirically find that during successful test-time adaptation (TTA) the variance of the gradients seem to follow a log-normal distribution (or rather that the log-normal distribution can fit the gradient variance well). They show in various experiments that departure from this empirical phenomenon correlates strongly with bad adaptation performance. Based on this insight, they then design an adaptive update rule that can be combined with other TTA methods in the literature to increase the adaptation performance further. The adaptive rule can scale the learning rate accordingly to improve adaptation stability. It is shown in various experiments that TTA performance improves when this adaptive rule is combined with existing state of the art TTA methods.", "questions": "- The paper does not concern foundation models in particular, so the mention of foundation models at the start of the abstract is misleading and inappropriate. Moreover, The refs in line 22 have nothing to do with foundation models, they are CNNs for image recognition, of course they can be used with/inside foundation models but foundation models typically mean (sometimes multi-modal) LLMs.\n- Model collapse is not defined in the introduction adequately I think. Performance degradation does mean model collapse (line 157), which rather means catastrophic forgetting of the source I believe.\n- The first paragraph seems to lump two different (but related) topics: domain adaptation (DA) and continual learning (CL). The references seem to be mainly from DA whereas the discussion mostly relates to CL (online ML better translates to CL, especially if the particular setting of unsupervised DA is not mentioned). The two fields have different emphasis: very roughly, CL focuses on catastrophic forgetting, and memory related issues while DA focuses on alignment of distributions to transfer well to unlabeled target distribution. Test-time UDA does focus also on the harder problem of how to align the distributions when the source inputs are not available. From the paragraph's discussion the 'collapse' seems to relate more to the 'catastrophic forgetting' studied in CL, rather than to the 'negative transfer' of UDA methods. Hence I would recommend clarifying the discussion or at least choosing the references more carefully.\n- The next paragraphs seem to clarify somewhat the above distinctions, however the confusion in the first paragraph can be alleviated by explicitly mentioning/separating the UDA/CL fields in 1st paragraph explicitly.\n- Section 2.1 is repetitive: I would recommend streamlining with section 1, as the content is discussed there already. \n- But otherwise, the paper would definitely benefit from a wider and more detailed related works section, some of which should be streamlined with the main text. For instance, there is a whole series of research on the 'heavy tail' phenomenon in deep learning theory and practice (e.g. SGD, Hessians, learned weights etc.), which is not mentioned at all. Were the authors not influenced by this research direction? Also, Bayesian deep learning and the parameter distributions that evolve during SGD-like updates are not mentioned adequately (except briefly in the appendix).\n- Theorem 1 does not seem new at all, moreover (6) is I believe incorrect: Sigma_{k|k-1} should have Sigma_{k-1} added to sigma_k^{2}\\deltat^{2}I. More importantly I believe, Theorem 1 could have been derived just from the discrete case by propagating the Gaussian parameters along the gradient descent step, so it is not clear why the authors present a continuous time formulation. The parametric distributions can be recovered using, e.g., Bayesian or approximate Bayes methods. For instance in variational Bayes approaches, a lot of works present optimizers that include a KL-divergence term as a regularizer (to prevent distributional collapse), and can then formulate various forms of parametric distribution updates.\n- It is not clear why we need the conjugate step in the algorithm presented in section 5.1. The align step already scales the learning rate\n  and hence changes the model f_k right? Why do you need the conjugate step? Do you have ablations that show that this step is needed?\n- In the experiments, it would help to explain the Correlated Label setting more clearly.\n- Would be good to vary the fixed sequence of domains in the experiments, has this been done? What is the sequence used to generate the data shown in Tables 1-3?\n- What is the model used in section 6? Add more details for the experiments and streamline such discussion with the main text.\n\nMinor comments:\n- In eq. (1) w_hat's dependence on t_{k-1} is confusing: in (2) we minimize that function w.r.t w_hat so t_{k-1} seems to be erroneous.", "limitations": "I think the authors could say more about the limitations of the paper, as of now only the Gaussian assumption of the parameter distribution as the limiting factor is mentioned in the appendix.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I liked the experimental nature of the paper and the results seem quite promising. However, as mentioned below, the paper would benefit from a more thorough related works section and also on the motivation of the heavy tail phenomenon. Also, the continuous time formulation presented in Theorem 1 does not seem to be particularly useful or relevant to the rest of the paper, which is experimental and not theoretical. See below for more detailed comments on these issues.", "was_revised": true, "final_justification": "I am copying my response to the AC's question above:\n\n1) Question of theoretical contribution: The authors do not claim to make a theoretical contribution, they clearly state throughout that the paper is based on an (interesting) empirical observation (of the relation between TTA performance stability and the inverse gamma distribution of the log variance). The Theorem 1 is there as a modeling framework to elucidate \"the unique relationship between the variance and time interval that emerges when discretizing a continuous-time stochastic differential equation\" (quoted from their rebuttal). I believe such results are common place in SGD literature (I would double check the references in the rebuttal [a]-[g] to be sure, if I had time). Perhaps the authors should tone down their statement of Theorem 1 to satisfy the AC.\n\n2) Acceptance of paper after extensive rebuttal: I believe the paper should be accepted in any case. I thought the experiments were extensive and based on my limited experience, I don't think that ImageNet-C is very narrow as an experimental setup. Note that (as stated also in the rebuttal by the authors) the paper includes many more experiments besides the ImageNet-C setup.", "rebuttal": "We sincerely thank the reviewer for the concrete and constructive feedback. We will revise the manuscript to incorporate your valuable suggestions, including Q1–Q5, Q9, and the Minor comments. Below are our detailed responses to your questions.\n\n---\n\n**W, Q6. For instance, there is a whole series of research on the 'heavy tail' phenomenon in deep learning theory … Were the authors not influenced by this research direction?** \n\nThank you for highlighting this important line of research. We provide a brief response here and will incorporate a more comprehensive discussion in the revised manuscript.\n\nA growing body of work has demonstrated that the noise introduced by SGD exhibits heavy-tailed behavior [a, b, c, d]. These studies suggest that the tail behavior of SGD noise is closely related to the flatness of loss minima and, by extension, generalization performance [b, c]. This theoretical understanding provides a foundation for the empirical observations reported in Appendix B.2 and supports our treatment of variance not as a fixed value but as a random variable.\n\nIn parallel, recent work has reinterpreted SGD-like updates from a Bayesian perspective. For instance, [e, f] model SGD as an SDE where the stationary distribution approximates a Gaussian posterior. This work offers a principled explanation for the implicit regularization effects of SGD. Building on this view, [g] have proposed posterior approximations using the trajectory of SGD itself. This work fits a Gaussian to the empirical mean and covariance of SGD iterates to enable practical Bayesian ensembling. These studies primarily focus on training from scratch or within supervised learning settings.\n\nIn contrast, our work extends this Bayesian viewpoint to the context of sequential domain shifts and unsupervised online adaptation, where the behavior of SGD dynamics is markedly different. Based on empirical evidence, we observe that the log-variance of parameter transitions aligns well with the IG distribution under stable conditions. Motivated by this observation, we propose SIGMA, an algorithm that maintains this alignment to achieve stable and flexible online adaptation.\n\nWe will include this discussion to strengthen the Significance of our contribution.\n\n---\n**W, Q7. … why the authors present a continuous time formulation. The parametric distributions can be recovered using, e.g., Bayesian or approximate Bayes methods.**\n\nThank you for your detailed observation. We address the two points you raised in turn, as outlined below.\n\n*Regarding Eq. (6)*: In general, $\\Sigma_{k|k-1}$ include the intermediate prior covariance $\\Sigma_{k-1}$. However, as stated in Lemma A.9 in Appendix A.1, we explicitly initialize the covariance as $\\boldsymbol{0}$. This assumption simplifies the analysis and leads to a more tractable update rule. We will make this modeling choice more explicit in the revised manuscript to improve clarity.\n\n*On approximate Bayesian methods and the motivation for the continuous-time formulation*: Beyond our use of the continuous-time formulation, it is possible to estimate the parameter distribution via approximate Bayesian methods. In particular, variational inference offers a principled approach to approximating the posterior using a chosen functional form, such as a Gaussian.\n\nHowever, our focus is not on modeling the full form of the parameter distribution, but rather on the unique relationship between the variance and time interval that emerges when discretizing a continuous-time stochastic differential equation. Specifically, we leverage the form $\\sigma_k^2 \\Delta t^2 \\mathbf{I}$, which naturally arises in our framework. This approach allows us to capture the evolution of the parameter distribution and regulate it through variance dynamics. Instead of fitting the parameter distribution directly, we concentrate on aligning the variance distribution with the IG form to guide adaptation.\n\nWhile selecting an appropriate proposal distribution for variational inference can be a viable direction for regularization, such approaches do not directly reveal the underlying dynamics of variance in online adaptation. Investigating alternative approximate Bayesian methods in this context is an interesting direction, and we leave it as a potential area for future work.\n\n---\n\n**Q8. The align step already scales the learning rate and hence changes the model f_k right? Why do you need the conjugate step? Do you have ablations that show that this step is needed?**\n\nIn continuation of our response to W, Q7: the conjugate step provides a mathematical foundation that aligns the transition distribution derived in Eq. (1) with the predictive distribution $p(y | \\boldsymbol{x}, \\boldsymbol{w})$.\n\nThe align step modifies only the parameter dynamics via the transition distribution. After accumulating this to form the parameter distribution, the conjugate step ensures consistency with the predictive distribution through Bayes' rule and a plug-in approximation, as shown in Eq. (13). This allows the predictive model to integrate the Gaussian mean as $f(\\boldsymbol{x}_k;\\boldsymbol{w})=f(\\boldsymbol{x};\\mu_k)$.\n\nAs you correctly intuited, the algorithm could function based solely on the align step, but the conjugate step provides the theoretical justification needed to apply this in a probabilistic optimization framework.\n\n---\n\n**Q10.  Would be good to vary the fixed sequence of domains in the experiments, has this been done? What is the sequence used to generate the data shown in Tables 1-3?**\n\nAs you correctly noted, we include results under four domain orders in Appendix D.3. Tables 1–3 report results using Order N →B →W →D, which is also used in Section 4.\n\nWe will explicitly state this in the main text to improve Clarity.\n\n---\n\n**Q11. What is the model used in section 6? Add more details for the experiments and streamline such discussion with the main text.**\n\nAs noted in Appendix C, we use a self-supervised version of the base ViT model as our default backbone.\n\nWe will incorporate this information and additional experimental details from Appendix C into the main text to enhance Clarity.\n\n---\n\n**References**\n\n[a] A tail-index analysis of stochastic gradient noise in deep neural networks. ICML 2019.\n\n[b] Hausdorff dimension, heavy tails, and generalization in neural networks. NeurIPS 2020.\n\n[c] The heavy-tail phenomenon in SGD. ICML 2021.\n\n[d] On the heavy-tailed theory of stochastic gradient descent for deep neural networks. arXiv preprint 2019.\n\n[e]  Stochastic Gradient Descent as Approximate Bayesian Inference. JMLR 2017.\n\n[f] Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks.  ITA 2018.\n\n[g] A Simple Baseline for Bayesian Uncertainty in Deep Learning. NeurIPS 2019."}, {"review_id": "Q7x8CMJ1t8", "reviewer": "Reviewer_TMwY", "summary": "The paper proposes a probabilistic framework that models the parameter adaptation for test-time adaptation (TTA) using stochastic differential equation (SDE). The authors show empirically that for stable adaptation, the log-variance distribution of the SDE's transition aligns close to an inverse-gamma (IG) distribution and that deviating from this alignment correlates strongly with model collapse.\nThus, they propose the Structured Inverse-Gamma Model Alignment (SIGMA) algorithm, based on this observation.\nSIGMA can be used with existing TTA approaches, and the authors have shown experimental results on the ImageNet-C dataset, showing improved performance for the proposed approach.", "questions": "1. The experiments are performed only for the ImageNet-C dataset benchmark. Whether the authors have performed experiments on other benchmarks such as CIFAR10C, CIFAR100C, etc.?\n2. How does the proposed approach perform in a more non-i.i.d. setting of continual TTA [1][2]? It is also noteworthy that [2] proposes a probabilistic framework, though it is different from the SDE-based probabilistic framework proposed in this paper.\n3. Could you provide experimental analysis showing how well the assumptions hold when there is a drastic distribution shift? \n4. Related to question 3, the continual TTA setting will be a more realistic test for this assumption, but more broadly, how well do the authors feel that the proposed approach will generalize to the continual TTA setting?\n\n**References**\n1. Wang, Qin, et al. \"Continual test-time domain adaptation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n2. Brahma, Dhanajit, and Piyush Rai. \"A probabilistic framework for lifelong test-time adaptation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "limitations": "The reliance of Theorem 1 on the strong assumption that both the gradient and the variance term remain approximately constant within each discrete time-step is a limitation that needs to be discussed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n* This paper focuses on the parameter dynamics using SDEs, which is different from existing works on TTA that focus on the loss function, parameter efficiency, pseudo-labeling, or sample selection.\n* The empirical observation that the log-variance of the SDE's transition aligns closely with an IG distribution is an interesting empirical result.\n\n**Weaknesses**\n* The experimental evaluation is limited. The experiments are performed only for the ImageNetC dataset benchmark, whereas existing approaches report their experiments on other benchmarks such as CIFAR10C, CIFAR100C classification, and/or Cityscapes-to-ACDC semantic segmentation.\n* Theorem 1 involves questionable assumptions. The paper's theoretical claim relies on Theorem 1, which is about the discretization of the SDE approximation. This theorem relies on the assumption that both the gradient and the variance term remain approximately constant within each discrete time-step (Appendix A.1, Line 494). This is a strong assumption in the context of the non-i.i.d. setting that the paper focuses on. When the model encounters an abrupt domain shift (say, moving from noise to blur), the loss landscape and, thus, the gradients can change dramatically. This paper does not show how well this assumption holds in practice.\n* The paper shows empirically that IG alignment correlates with stability, but there is no theory on the reason for this correlation. Is the correlation specific to a particular dataset, ImageNetC? Results on more datasets will help in supporting the claims made in the paper.", "was_revised": true, "final_justification": "The authors have addressed most of my queries.\n\nThey have reported additional comparisons with other recent CTTA approaches during the discussion.\n\nHowever, these comparisons should be included in the main paper, and more discussions on related works, compared state-of-the-art approaches, and benchmark datasets should be provided.\n\nThus, I will keep my score unchanged.", "rebuttal": "We sincerely thank the reviewer for the insightful and thorough feedback. We will revise the manuscript to incorporate your valuable comments. Below, we provide detailed responses to your questions.\n\n---\n\n**W1, Q1. The experiments are performed only for the ImageNet-C dataset benchmark. Whether the authors have performed experiments on other benchmarks such as CIFAR10C, CIFAR100C, etc.?** \n\nRecent works [a, b, c, d] have primarily focused on more realistic, large-scale datasets such as ImageNet-C, which contain a higher number of classes and are more challenging. Nonetheless, we have evaluated our method on a variety of datasets, including D109, Sketch, and Rendition, as shown in Appendix D.6. Furthermore, in Appendix D.7, we extend our method to a speech recognition task in a continual multi-speaker adaptation scenario using realistic TED talk recordings.\n\nWe have also conducted experiments on CIFAR10C and CIFAR100C using ResNet-based models. Below are the results of applying SIGMA to both TENT and ROID baselines:\n\n| Dataset | Source | TENT | +SIGMA | ROID | +SIGMA |\n| --- | --- | --- | --- | --- | --- |\n| CIFAR10C | 43.5 | 19.8±0.07 | 18.2±0.06 | 16.3±0.17 | 15.7±0.06 |\n| CIFAR100C | 46.4 | 32.5±0.07 | 31.2±0.20 | 31.7±0.11 | 29.6±0.07 |\n\nAs shown, our method consistently improves performance on both datasets across different baselines, including the standard method TENT and the stronger baseline ROID.\n\nWe will explicitly include this explanation to improve the Quality of the paper.\n\n---\n\n**W2, Limitations. This theorem relies on the assumption that both the gradient and the variance term remain approximately constant within each discrete time-step.**\n\n**Q3. Could you provide experimental analysis showing how well the assumptions hold when there is a drastic distribution shift?**\n\nThank you for the insightful comment. This assumption is naturally satisfied in the online adaptation setting for the following reasons.\n\nAs explained in Appendix A, our derivation assumes the gradient and variance are approximately constant during each discretization step. This assumption is justified by the small time interval involved in transitioning from continuous to discrete time. \n\nThe time interval is determined by the learning rate $\\eta$, and we adopt a learning rate in the range of $10^{-5}$ to $10^{-6}$—which is about 100 times smaller than that used during source model training (typically $10^{-3}$ to $10^{-4}$). Empirically, as shown in Figure 6 (x-axis), the maximum observed variance is around $e^{-11}$ and the minimum is approximately $e^{-28}$, demonstrating that variance remains extremely small on continual TTA settings. This behavior is consistent across multiple datasets, as shown in Figure 15. \n\nThe reason for this consistency is that TTA assumes a well-trained model as its starting point (Section 2.1), leading to minimal model drift during adaptation. Consequently, the assumption of approximately constant mean and variance is naturally satisfied in the TTA setting.\n\nWe will explicitly include this explanation to improve the Quality of the paper.\n\n---\n\n**W3. Is the correlation specific to a particular dataset, ImageNetC? Results on more datasets will help in supporting the claims made in the paper.**\n\nFigure 15 provides validation of the IG distribution assumption across various datasets. Additionally, the results shown in Figure 10, which include diverse model architectures, further confirm the robustness of the observed pattern.\n\n---\n\n**Q2. How does the proposed approach perform in a more non-i.i.d. setting of continual TTA [1][2]?**  \n\nAll experiments, including the IG distribution analysis in Section 4, are conducted under a continual TTA setting—i.e., we do not reset the parameters between adaptation episodes. Moreover, Appendix D.3 includes experiments that vary the domain order to test robustness. \n\nTables 2, 3, and 8 further evaluate performance under more challenging non-i.i.d. scenarios, such as Label Shift.\n\nWe will emphasize this setup more clearly in Section 6 to improve Clarity.\n\n---\n\n**Q4. Related to question 3, the continual TTA setting will be a more realistic test for this assumption, but more broadly, how well do the authors feel that the proposed approach will generalize to the continual TTA setting?**\n\nThank you for the constructive question. All of our experiments were conducted under continual TTA settings. In Appendix D, we further extend our evaluation to test generalizability across diverse model architectures (Appendix D.1), realistic scenarios such as domain order variation and long streams (Appendix D.3 and D.5), and a variety of datasets, including D109, Rendition, and Sketch (Appendix D.6).\n\nTo explore its applicability in real-world settings, we also apply our algorithm to a speech recognition task involving continual multi-speaker adaptation using realistic TED talk recordings (Appendix D.7).\n\nAcross all these experiments, SIGMA consistently improves performance across a wide range of settings. These results support our central claim: enforcing the IG alignment in parameter dynamics helps prevent collapse and leads to robust performance in both online and continual adaptation settings.\n\n---\n\n**References**\n\n[a] Towards stable test-time adaptation in dynamic wild world. ICLR 2023.\n\n[b] Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors. ICLR 2024.\n\n[c] Continual momentum filtering on parameter space for online test-time adaptation. ICLR 2024.\n\n[d] Test-Time Model Adaptation with Only Forward Passes. ICML 2024."}, {"review_id": "sJxmNSJufO", "reviewer": "Reviewer_ryf8", "summary": "This paper investigates the parameter dynamics of online machine learning and its application in test-time adaptation (TTA). The key findings include: (1) the fitness of inverse gamma (IG) and log-variance serves as a strong predictor of adaptation performance and model collapse, and (2) state-of-the-art TTA methods often implicitly promote IG alignment. Furthermore, the authors propose SIGMA, a method that dynamically estimates an appropriate IG distribution using derivative-free optimization and adjusts the parameter update trajectory to maintain alignment during online adaptation. Experimental results demonstrate the effectiveness of SIGMA.", "questions": "* In line 511, authors show that DUC dataset is used. However, the motivation and details of DUC is not introduced and not any reference. DomainNet-126 [3] , a subsect of Domain Net[2] , or full DomainNet is more widely used in related papers [1,4].\n\n* Some references should be cited with published format, such as [40], please check all references.\n \n* How to ensure $c_k \\leq 0$ and $b_k > 1$ (line 244)? use Lagrange multiplier or just clip?\n* How to understand alignment length $\\lambda$ (in line 245)? \n* It is better to show how to solve Eq.(9) and get Eq.(10). And how to get $r_k$ and $T_k$ ?\n\n**Reference**\n1. Robust Test-Time Adaptation in Dynamic Scenarios. CVPR 2023\n2. Moment Matching for Multi-Source Domain Adaptation. ICCV 2019.\n3. Semi-supervised Domain Adaptation via Minimax Entropy. ICCV 2019.\n4. Feature alignment and uniformity for test time adaptation. CVPR 2023.", "limitations": "Please see Questions.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n* This paper studies the fundamental and important question about parameter dynamics in online machine learning and test-time adaptation.\n\n* Interesting and novel findings about parameter dynamics.\n\n* A plug-and-play and effective method improves the performance of TTA methods.\n\n**Weaknesses**  \nI do not find any major weakness. However, I have some questions and advise, see below.", "was_revised": true, "final_justification": "I have carefully reviewed the authors' rebuttal. My concerns have been addressed, and I will keep my score (5, accept).", "rebuttal": "We thank the reviewer for the detailed comments. We will revise the manuscript to incorporate your valuable suggestions (Q1 and Q2). Below are our responses to your additional questions.\n\n---\n\n**Q3. How to ensure $c_k\\leq1$ and $b_k>1$ (line 244)? use Lagrange multiplier or just clip?**\n\nThank you for the insightful question. As you correctly noted, we solve for the optimal values within the feasible domain and apply clipping if the solution lies outside the valid region. This provides a simple and effective way to enforce the constraints. In practice, we implement this using maximum likelihood estimation with a simplex-based optimizer that handles constraints efficiently.\n\nWe will incorporate this explanation into Section 5.\n\n---\n\n**Q4. How to understand alignment length $\\lambda$ (in line 245)?**\n\nThe alignment strength $\\lambda$ quantifies how closely the updated distribution should follow the real-time reference distribution $q$, which is assumed to follow the IG form. A value of $\\lambda=0$ implies complete distrust in the reference distribution, effectively setting $\\Delta t_k=0$, which freezes parameter updates. Conversely, higher values of $\\lambda$ indicate greater trust in the reference distribution and thus allow larger parameter updates.\n\nWe will incorporate this explanation into Section 5.\n\n---\n\n**Q5. how to solve Eq.(9) and get Eq.(10). And how to get $r_k$ and $T_k$?**\n\nAs noted in Q3, we obtain the $c_k$ and $b_k$ from Eq. (9) within the constrained domain. Equation (10) is derived from Eq. (7):\n\n$\n\\tilde{v}_k = (v_k - c_k)/b_k + 2\\log \\lambda\n$\n\nwhere $v_k = \\log(\\sigma_k^2 \\Delta t^2) = 2\\log(\\sigma_k \\Delta t)$. Substituting and simplifying, we get:\n\n$\n\\frac{\\tilde v_k}{2} = \\log \\lambda + \\frac{\\log(\\sigma_k \\Delta t) - \\tfrac{1}{2}c_k}{b_k}.\n$\n\nTaking the exponential of both sides:\n\n$\n\\exp\\left(\\frac{\\tilde v_k}{2}\\right)\n= \\lambda \\exp\\left( -\\frac{c_k}{2b_k} \\right)\\exp\\left( \\frac{\\log(\\sigma_k \\Delta t)}{b_k} \\right)=\\lambda r_k T_k \\sigma_k \\Delta t,\n$\n\nwhere $\\exp\\left( \\frac{\\log(\\sigma_k \\Delta t)}{b_k} \\right) = (\\sigma_k \\Delta t)^{\\frac{1 - b_k}{b_k}}\n(\\sigma_k \\Delta t) = T_k (\\sigma_k \\Delta t)$.\n\nTaking the logarithm of both sides:\n\n$\n\\tilde v_k = 2\\log |\\sigma_k (\\lambda r_k T_k) \\Delta t|.\n$\n\nThis derivation provides a complete and interpretable expression for $\\hat{v}_k$, connecting it back to the dynamics of the variance term and the alignment process.\n\nWe will add this derivation to the manuscript for completeness."}, {"review_id": "0t3hkS4OXR", "reviewer": "Reviewer_Popp", "summary": "The paper models the dynamics of weights during online learning and test-time adaptation as a stochastic differential equation. It uses a discretization to obtain the transition distribution of the parameter evolution. The paper focuses on the transition variance and argues that the log-variance of the transition distribution follows an inverse gamma distribution when adaptation is stable. Based on this argument, the paper proposes an unsupervised adaptation mechanism that aligns the log-variance to an inverse gamma distribution to enable stable adaptation to a non-stationary test stream.", "questions": "- Equation 13 suggests a Bayesian treatment of the parameters. This makes sense, given that the paper focuses on the parameters' variance. However, from the text, it seems that the method is just evaluated at the mean. Could we see results with Monte Carlo sampling as well, incorporating the adjusted variance term?\n\n- How come the adaptation gains of most TTA baselines in the standard setting of Table 1 are close to zero (e.g., for TENT, SAR, RoTTA, EATA)? These results seem to deviate from other papers \\[1, 2], where the adaptation gains on ImageNet-C severity 5 are quite pronounced.\n\n[1] Niu, Shuaicheng, et al. \"Efficient test-time model adaptation without forgetting.\" International Conference on Machine Learning. PMLR, 2022.\n\n[2] Zhao, Hao, et al. \"On pitfalls of test-time adaptation.\" ICML (2023).", "limitations": "Limitations are not discussed.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n\n- The continuous-time SDE formulation of SGD updates in online settings, while not novel [3], seems an interesting and suitable theoretical framework for analyzing dynamic adaptation methods.\n- The paper is clearly written and easy to follow.\n\n# Weaknesses\n\n- Most related work is not (sufficiently) discussed. The following papers also model the dynamics of weights in TTA:\n1. Lee, Jae-Hong, and Joon-Hyuk Chang. \"Stationary latent weight inference for unreliable observations from online test-time adaptation.\" ICML 2024.\n\n2. Lee, Jae-Hong, and Joon-Hyuk Chang. \"Continual momentum filtering on parameter space for online test-time adaptation.\" ICLR 2024.\n\n3. Lee, Jae-Hong. “Bayesian Weight Enhancement with Steady-State Adaptation for Test-time Adaptation in Dynamic Environments,” ICML 2025.\n\nIn particular, 3.) also uses an SDE formulation for parameter dynamics and adjusts the step size $\\Delta t$ based on the covariance. Could we see a detailed qualitative and empirical comparison against 3.)? As the paper is very similar both conceptually as well as in the experiments to 3.), yet not cited, could the authors please highlight how the current work differs from or extends 3.)?\n\n- The authors empirically find that the log-variance follows an inverse gamma (IG) distribution in stable adaptation settings and, in a second step, regularize the updates to follow an IG distribution. However, except for empirical correlation, the paper does not explain why the IG is the “holy grail.” What properties of the distribution explain this correlation? What is the intuition or theoretical connection behind this finding?\n\n- If I understand correctly, adapting the model results in modifying the time discretization $\\Delta t$. Does this correspond to changing the learning rate? If so, does this mean only the step size is changed but not necessarily the direction of the gradient?", "was_revised": true, "final_justification": "As stated in the response to the rebuttal, I think the paper needs major changes to (a) incorporate related work more thoroughly and (b) provide a better theoretical justification/explanation on why the core part of the method (IG alignment) works so well. In my opinion this should be a core part of the paper, but is currently missing entirely. I rate both of these concern as severe points and therefore believe the paper does not pass the bar for acceptance.", "rebuttal": "We sincerely thank the reviewer for the detailed feedback and insightful comments. We will revise the manuscript accordingly to reflect your valuable suggestions. Below, we provide our point-by-point responses to your questions.\n\n---\n\n**W-Q1. Could the authors please highlight how the current work differs from or extends 3 ?**\n\nThank you for recommending recent relevant studies. Below we provide a brief explanation and will include a broader and more detailed discussion in the revised manuscript.\n\nExisting works based on Bayesian deep learning primarily focus on Bayesian filtering [1.),2.),3.)]. The most recent work [3.)], in particular, emphasizes tracking the evolving parameter distribution, treating the variance as a simple variable and forcing it to converge to a fixed value. However, this approach tends to suppress the heavy-tail phenomenon excessively, which may limit adaptability [a,b,c,d]. Additionally, since these methods introduce latent parameters for filtering, they incur memory overhead comparable to the number of learnable parameters.\n\nIn contrast, our work treats variance as a random variable and jointly considers both the variance distribution and the parameter distribution. This enables us to explore the intrinsic properties of the parameter distribution and empirically demonstrate that performance improves when the variance follows the IG distribution (Section 4). We further propose an algorithm that estimates the IG distribution in real time and adjusts the parameter distribution accordingly (Section 5). This allows us to regulate variance without fixing it to a specific value, avoiding excessive suppression of heavy tails and achieving superior performance. Moreover, since we do not introduce latent parameters, our method is memory-efficient.\n\nWe will incorporate this discussion to strengthen the Originality and Significance of our contribution.\n\n---\n\n**W-Q2. What properties of the distribution explain this correlation? What is the intuition or theoretical connection behind this finding?**\n\nTheoretically, the IG distribution is the conjugate prior for the variance of a Gaussian likelihood. However, its heavy-tailed nature makes empirical analysis challenging (Section 3.2, Appendix E). To observe heavy-tailed behavior, we conduct analysis and alignment in log-scale (Sections 4 and 5).\n\nWe will add this explanation to improve the Clarity of our presentation.\n\n---\n\n**W-Q3. Does this correspond to changing the learning rate? If so, does this mean only the step size is changed but not necessarily the direction of the gradient?**\n\nThank you for this insightful question. Our IG alignment strategy derives a simple mechanism for adjusting the time interval when projecting from continuous-time to discrete-time space (Section 5 and Appendix B). This can be interpreted as an automatic, environment-specific dynamic adjustment of the learning rate without recomputing the gradient.\n\nCompared to methods like SAR, which explicitly consider the gradient direction, our mechanism consistently achieves higher performance (Tables 1–5). Furthermore, this mechanism contributes to the computational efficiency of our algorithm, which is crucial for online machine learning (Appendix D.2).\n\n---\n\n**Q1. Could we see results with Monte Carlo sampling as well, incorporating the adjusted variance term?**\n\nThank you for raising this important point. We conducted Monte Carlo sampling over 10 iterations and present the results below for the ROID-based SIGMA experiments under the conditions in Tables 1–3:\n\n|  | Correlated Input | Correlated Label ($\\gamma$ = 0.1) | Correlated Label ($\\gamma$ = 0.0) |\n| --- | --- | --- | --- |\n| Plug-in Approximation in Eq. (13) | 41.8±0.14 | 37.7±0.01 | 39.8±0.05 |\n| Monte Carlo Sampling | 41.6±0.13 | 37.6±0.04 | 39.8±0.05  |\n\nWhile Monte Carlo sampling led to marginal improvements, the difference was minimal. This is likely because the model is already well-trained, resulting in only minor changes during adaptation. Indeed, the mean of the variance remains very small (between $e^{-21}$ and $e^{-23}$) as shown in Figure 6(d).\n\nMoreover, Monte Carlo sampling requires repeated predictions, increasing inference time by a factor of 10, which significantly impacts efficiency in online learning. This result supports our decision to use the plug-in approximation over Monte Carlo sampling.\n\nWe will include this discussion to enhance the Significance of our work.\n\n---\n\n**Q2. How come the adaptation gains of most TTA baselines in the standard setting of Table 1 are close to zero (e.g., for TENT, SAR, RoTTA, EATA)?**\n\nWe strictly followed the experimental protocol of the latest open-source standard TTA benchmark (Section 6 and Appendix C). Our implementation uses self-supervised ViT models, consistent with those in recent literatures [1.),2.),3.)], and the baseline trends we observe are similar.\n\nOn the other hand, [1,2] primarily use less powerful ResNet or pure ViT backbones. We have also evaluated our method on both pure ViT and Swin architectures, as reported in Tables 4 and 5, and obtained results that align closely with the findings in [1,2].\n\n**References**\n\n[a] A tail-index analysis of stochastic gradient noise in deep neural networks. ICML 2019.\n\n[b] Hausdorff dimension, heavy tails, and generalization in neural networks. NeurIPS 2020.\n\n[c] The heavy-tail phenomenon in SGD. ICML 2021.\n\n[d] On the heavy-tailed theory of stochastic gradient descent for deep neural networks. arXiv preprint 2019."}]

Author rebuttal:

Meta-review: The authors study learning dynamics during test-time adaptation, and empirically find that during successful test-time adaptation (TTA), the variance of the gradients follows a log-normal distribution. From here, the authors analyse that deviations from this empirically observed "sweet spot", model performance degrades (in TTA, often referenced to as model collapse) -- as the authors note, "The log-variance portrait thus provides a statistically grounded diagnostic signal for evaluating adaptation quality". This phenomenon is validated across artificially generated noise domains (Noise, Blur, Weather, and Digital). The authors consider some (DeYO, ROID) methods for sample-filtering and note that these methods incourage to stay close to the desired distributional properties.

Based on this observation, the authors propose an algorithm, SIGMA, which is an addition to existing test-time adaptation algorithms and aids with stable adaptations in scenarios where the algorithms would otherwise collapse. On ImageNet-C, the authors demonstrates that SIGMA is able to improve the performance of various methods and rescues the performance of TENT which would otherwise fail to adapt beyond the baseline level. For longer term adaptation, SIGMA can complement approaches like ROID and improves longer term adaptation performance on repeated ImageNet-C.

Reviewers recommend acceptance based on the interesting and convincing empirical observation and study of IG distributed variance (ePCh, TMwY, ryf8). Reviewer Popp raised justified concerns about the positioning within the literature studying learning dynamics of TTA, and the authors provided a suggested revision to their paper in the discussion phase. Concerns that the paper needs to provide more theoretical motivation were discussed and reviewers did not reach an agreement on whether additional theoretical justification is required.

Based on the discussions, and my own reading of the manuscript, I believe that this paper is a great example how an empirically and rigorously tested finding about model behavior can guide algorithm design. While I agree with reviewer Popp that additional theoretical investigation and and explanation for this behavior would be a valuable addition, such a theoretical investigation is beyond the scope of the current study (and would have made the difference to proposing the paper as a spotlight). 

In general, I recommend acceptance, provided the authors make the following updates to their manuscript:

- It should be very clear that the paper discovers an empirical law and uses this to build an adaptation method. The authors might consider to tune down the presentation of *Theorem* 1, which on the first read seems misleading and like a central piece of the work, while it is in fact mostly incremental to Särkkä & Svensson [50]. This could be stressed more, also in the main paper, to clarify the contribution.
- Comparisons to additional state of the art methods
- More discussion on related work as provided during the rebuttal
- Additional benchmark datasets as provided during the rebuttal
Decision: Accept (poster)