Un-Mixing Test-Time Normalization Statistics: Combatting Label Temporal Correlation
Authors: Devavrat Tomar
Guillaume Vray
Jean-Philippe Thiran
Behzad Bozorgtabar
Venue: ICLR 2024 poster (2024)
Keywords: test-time adaptation
batch normalization
distribution shift
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose a new test-time normalization layer to combat label temporal correlation.

Abstract: Recent test-time adaptation methods heavily rely on nuanced adjustments of batch normalization (BN) parameters. However, one critical assumption often goes overlooked: that of independently and identically distributed (i.i.d.) test batches with respect to unknown labels.  This oversight leads to skewed BN statistics and undermines the reliability of the model under non-i.i.d. scenarios. To tackle this challenge, this paper presents a novel method termed '$\textbf{Un-Mix}$ing $\textbf{T}$est-Time $\textbf{N}$ormalization $\textbf{S}$tatistics' (UnMix-TNS). Our method re-calibrates the statistics for each instance within a test batch by $\textit{mixing}$ it with multiple distinct statistics components, thus inherently simulating the i.i.d. scenario. The core of this method hinges on a distinctive online $\textit{unmixing}$ procedure that continuously updates these statistics components by incorporating the most similar instances from new test batches. Remarkably generic in its design, UnMix-TNS seamlessly integrates with a wide range of leading test-time adaptation methods and pre-trained architectures equipped with BN layers. Empirical evaluations corroborate the robustness of UnMix-TNS under varied scenarios—ranging from single to continual and mixed domain shifts, particularly excelling with temporally correlated test data and corrupted non-i.i.d. real-world streams. This adaptability is maintained even with very small batch sizes or single instances. Our results highlight UnMix-TNS's capacity to markedly enhance stability and performance across various benchmarks. Our code is publicly available at https://github.com/devavratTomar/unmixtns.

Published as a conference paper at ICLR 2024
UN- UN-MIXING TEST-TIME NORMALIZATION STATISTICS: COMBATTING LABEL TEMPORAL CORRELATION
Devavrat Tomar
Guillaume Vray?- Jean-Philippe Thiran 1.2
Behzad Bozorgtabar
'EPFL CCHUV
{firstname)- {last mastnameeeepfL.cco
ABSTRACT
Recent test-time adaptation methods heavily rely on nuanced adjustments of batch normalization (BN) parameters. However, one critical assumption often goes overlooked: that of independently and identically distributed (i.i.d.) test batches with respect to unknown labels. This oversight Icads to skewed BN statisties and undermines the reliability the model under non-i.i.d. scenarios. To tackle this challenge, this paper presents novel method termed 'Un-Mixing Test-Time Normalization Statistics' (UnMix-TNS) Our method re-calibrates the statistics for each instance within a test batch by mixing it with multiple distinct statisties compo- nents, thus inherently simulating the i.i.d. scenario. The core of this method hinges on distinctive online unmixing procedure that continuously updates these statistics components by incorporating the most similar instances from new test batches. Re- markably generic its design, UnMix-TNS seamlessly integrates with a wide range of leading test-time adaptation methods and pre-trained architectures equipped with BN layers. Empirical evaluations corroborate the robustness of LnMix--TN under varied scenarios-rangee from single to continual and mixed domain shifts, particularly excelling with temporally correlated test data and corrupted i.d. real-world streams. This adaptability is maintained even with very small batch sizes or single instances. Our results highlight UnMix-TNN TNS's capacity to markedly enhance stability and performance across various benchmarks. Our code is publicly available at https: / / thub. com/ devdvevaarree oma r /ummi xtns.
INTRODUCTION
Deep neural networks (DNNs), when deployed in real-world test environments, often face the pervasive challenge of domain shift, potentially undermining their performance. Addressing this, the research community has advanced towards the forefront of online test-time adaptation (TTA). This involves a myriad of methodologies aimed at recalibrating the batch normalization (BN) layers (Toffe Szegedy 2015). cornerstone of deep learning architectures informed by real-time test data, BN layers play critical role in stabilizing the training process and enhancing model generalization. These TTA approaches function by either re-estimating normalization statistics based on the present test batch (Nado et al.) 2020 Schneider al. 2020 Benz al. 2021) additionally fine-tuning the BN parameters to minimize test-time losses, such as those resulting from entropy minimization (Wang al. 2020 Specifically, the former has concentrated
addressing the performance decline observed in conventional BN when subjected to domain shifts during testing. This diminution in model efficacy on previously unseen test data is primarily ascribed to shifts in the statistical properties of intermediate layers relative to those conserved from the source training dataset. Intrinsically, BN marginalizes inconsequential instance-wise variations by decorrelating feature sets across batches, assuming that these batches are uniformly populated with samples from diverse categories. If test batches are also uniformly sampled from different categories, employing TTA methods that renormalize features based on immediate statistics from the current test batch can counteract the donain-induced distribution shifts.
Nevertheless, these methods come with their own set of challenges and assumptions. Typically, they operate under the assumption of large test batch sizes and a singular, static distribution shift.
denotes equal contribution.

---

Published as a conference paper at ICLR 2024
Moreover, these methods generally presume that the test batches are independently and identically distributed (i.i.d.) concerning their true labels. This i..d. assumption, though useful for simplification, frequently does not hold true under real-world scenarios. Take the case of autonomous driving, where a variety of diverse and unpredictable factors makes it improbable for incoming test batches to conform an i.i.d. distribution. In such contexts, conventional BN-based TTA methods fall short, producing unstable and unreliable adaptations. To overcome this issue, recent methods (Gong et al., memory
2022; Yuan et al., 2023 have introduced the concept of a balanced, pseudo-label-bane bank. This memory bank serves as a repository for test images, facilitating the online estimation of unbiased BN statisties and the integration of instance-level feature statistics with those derived from source data. While promising, their utility is often limited to particular situations. Notably, they falter scenarios where privacy concerns curtail data retention. Furthermore, choosing the optimal weight hyperparameter for merging instance-level statistics with pre-existing batch statistics can introduce layer of computational overhead wwwt-deploymeeeeee
complicating the model's adaptability. In contrast, methodologies such as (Niu al. 2023) and (Marsden et al. (2024) have advocated for instance-level normalization techniques, such as (Ba et al. 2016) as viable alternatives to BN for training on source data. Other alternatives exhibit increased robustness varying batch size and long-tailed distribution via group normalization (Wu & He] 2018), Compound Batch Normalization (Cheng et al. 2022). and
Mixture Normalization (Kalayeh & Shah 2019).
In this paper, we thoroughly revisit BN for test-time adaptation, targeting temporally correlated (non-i.id.)) distributionally shifted test batches. In our approach, we interpret the instance-wise input features of BN layers pertaining to the present test batch as samples drawn in non-i..d. manner from K distinct distributions over time, reflecting temporal correlation. Consequently, we wimix the initially stored batch normalization statistics into K distinctive components, each reflecting statistics from similar test inputs. This unveils a strategy for ral-time adaptation of these statistics to the streaming test batches. Drawing inspiration from sequential clustering paradigms, our method aims to update the K statistics components using the closest instances from the streaming test data in a dynamic online setting. In summary, contributions are follows:
We introduce novel test-time normalization scheme (UnMix-TNS) designed withstand the challenges posed by label temporal correlation of test data streams.
UnMix-TNS demonstrates robustness across various test-time distribution shifts such as single domain, and continual domain. While not primarily designed for mixed domain settings, it offers a level of adaptability in these scenarios. Additionally, the method excels with small batch sizes, even down to individual samples.
UnMix-TNS layers, with negligible inference latency, seamlessly integrate into pre-trained neural network architectures, fortified with BN layers, boosting test-time adaptation capabilities while incurring minimal overhead.
Our results demonstrate notable enhancements in TTA methods under non-i.i.d. conditions when augmented with UnMix-TNS, as evidenced on datasets involving corruptions and natural shifts. We also unveil ImageNet- VID-C and LaSOT-C video datasets, corrupted versions of ImageNet-VII and LaSOT, for realistic domain shift analysis in video classification.
2 METHODOLOGGY
2.1 PRELIMINARIES
Batch normalization in TTA. In addressing the challenges of covariate shifts at test time, test- time BN (TBN) employs the current batch's statistics rather than relying on stored source statistics. Consider a batch of feature maps being input into the BN layer. We can represent this batch as 2 E BxCxL where B stands for the batch size, C' signifies the number of channels, and L denotes the dimensions of each feature map. The BN layer normalizes the feature maps using the current batch statistics, denoted as (μ², o²) R Following this, it employs the affine parameters (y, RC to produce normalized feature maps à that are both scaled and shifted, as follows:
L 2,00 7o
Zee.: Hle + Bo,
Σ Zb,ad b.
Σ b.1
(Zb.e.t 1t)2, (I)
BL
BBL
2

---

Published as a conference paper at ICLR 2024
time (t) .
1-000
c-1000
c-1500
1-00
Text Dister tuution
(w) tax (4.4.d. setting)
Ingeea
dechcres
(b) T2S (aca-1.1.d. setting)
lacal srattacice
(c) UnMix-Ti TSE (000-1.1.4 1..4. octting)
Figure 1: Test-Time BN (TBN) vs. UnMix-TNS, (a) TBN recalibrates its intermediate features when test batches are i.i.d. sampled over time I, accommodating distribution shifts. (b) However, TBN fails for non-i.i.d. label-based test batch sampling, leading to skewed batch statistics. (c) UnMix-TNS overcomes this failure by estimating unbiased batch statistics through its K statistics components.
Test-time adaptation under label temporal correlation. Let fo X - y' denote a neural network parameterized by 0, mapping image space X' to label space. y. This network, featuring BN layers, has been trained on the source data distribution Ps(x. y). Given stream of covariate shifted test images X2 sampled at time é from arbitrary test distribution P7 y/t) with temporally correlated labels, the goal is to continuously adapt fo to new data Xx, as it arrives, even without access to the corresponding true label ye-
2.2 UNMIXING TEST-TIME NORMALIZATION STATISTICS
In Figure toy example clearly elucidates a crucial point: when dealing with covariate-shifted target test images that bear temporal correlations, there's an intrinsic bias in estimating online batch normalization statistics, depicted in Figure [b). This bias is in sharp contrast to the more stable dynamics of batch normalization statistics sourced from i..d. batches of well-shuflled data, as visualized in Figure aa). This variance can lead to substantial failures in many test-time adaptation methods, particularly when test images from the target domain exhibit temporal correlations tied to their true labels.
This section introduces UnMix-TNN TNS, our proposed normalization paradigm tailored for the non-i.i.d streams of test images. As illustrated in Figure [c). at the heart of the UnMix-TNS layer is process that deftly decomposes the stored BN statistics into K components. Then, all statistics components are updated based on their alignment with the test batch's instance-wise statistics. Components that align more closely undergo more substantial updates, while others reflect statistics from previously encountered, less similar features, simulating an ideal i.i.d. environment. The genius lies in the end result: the statistics computed by UnMix- TNSS with its K components in non-i.i.d. conditions, align seamlessly with those generated by TBN in i.i.d. conditions (see Appendix A.3 for theoretical analysis).
In the subsequent Section 2.2.1 we formulate the distribution of K UnMix-TNS components and describe how to compute the label unbiased normalization statistics at temporal instance f, utilizing only the statistics derived from the K components conjunction with the current statistics the non- i..dd test batch. In Section 2.2.2 explain the process of initializing the K statistics components of the UnMix-TNS layer, leveraging the batch normalization statistics stored within the provided pre-trained model. In Section 2,2 we outline the methodology for deriving current statistics from these K statistics components. This process is crucial for both normalizing the input test features and their dynamic online adaptation.
3

---

Published as a conference paper at ICLR 2024
Camporacce
lipet
Mixing Statiistice
(1
P
Update UnMt--11 Corgonanta
( of
(ikk ok)
7:19
yu,
Kla UELLLA 185
Link og
UnMixx-
grutsion
Figure 2: An Overview of UnMix- Given batch of non-i.id test features z' E BB at temporal instance f, we mix the instance-wir statistics (ā (f))a)) E R _BxC with K UnMix-TNS components. The alignment of each sample in the batch with the UnMix-TNS components is quantified through similarity-derived assignment probabilities pI. This aids both the mixing process and subsequent component updates for time + 1.
2.2.1 DISTRIBUTION OF K UNMIX-TNS COMPONENTS MIXTURE
Let [uf..... pkl and .ei......'k denote the statistics of the K components within the UnMix-TNS layer at given temporal instance t, where each p of E RRC, We articulate the distribution hy (z) of instance-wise test features E Eº marginalized over all labels at temporal instance using the K components:
(222 = 1 *NNCuiee ok.) K
(2)
where N represents the Normal distribution function. Given only (ut: of) one can derive label unbiased normalization test statistics (#* ; σ²) at time (see Appendix A.1 follows:
jt = =Eny [2e]
Euho
(3)
K
(0¹)² = Eng, [[ze- - 2'
K E
(hhe))
( Σ pk.0)
(4)
K
Subsequent sections will delve into the initialization scheme for the UnMix-TNS components and elucidate the process of updating their individual statistics of) at temporal instance t.
2.2.2 INITIALIZING UNMIX- TNS COMPONENTS
Consider the pair (n, a) E CRC which stands as the stored means and standard deviation of the features in given BN layer of fe. derived from the source training dataset before adaptation. We initialize statistics (Mî. of) of K components of UnMix-TNN TNS at = 0, as delineated in Equation 5)
K
Mk.c = Mc + Jet
K-1 Sk.e.
ohe = /1 a a Sk.e NO,
(5)
K
where Çe.e is sampled from the standard normal distribution AN 1), and (0, 1) is hyperpa- rameter that controls the extent to which the means of the UnMix-TNS components deviate from the stored means of BN layer during initialization. Note that the initial normalization statistics (m"
estimated from (ng. of) have an expected value equal to the stored statistics of the BN layer, i.c.,
Elz2] = μı and E. [ o² for all values of a. Further insights into this relationship are detailed in Appendix A.2
2.2.3 REDEFINING FEATURE NORMALIZATION THROUGH UNMIX-TNS
Considering the current batch of temporally correlated features E R BxCxL, we commence by calculating the instance- wise means and standard deviation (ft ,0) E R BxC, mirroring the Instance Normalization Ulyanov et al. 2016). i.e.:
1 /
Th. 7 Eehen
then Tbec-
D(ehrc (zheet - 11)
(6)
4

---

Published as a conference paper at ICLR 2024
To gauge the likeness between UnMix-TNS components and current test features, compute the cosine similarity $b, of the current instance-wise means P, with that of K BN components { ) follows:
5 - sim (16,)14))
(7)
where sim(u, u, v) = u'v
|i]|lvvl denote the dot product between l2 normalized u and v E RR We
proceed to derive the refined feature statistics, denoted as (a', _BCC for each instance. This accomplished by weighted mixing of the current instance statistics, (ă',ō') harmoniously blended with the K BN statistics components, (M oikke This mixing strategy unfolds similar to Equation (3) as:
1 ph. Mlyke
(8)
K
(othe)² 1 E(bh.
1 (phhrsr)) - Σ phu
(9)
K
ph = (1 pb. 10 + Pb, 1b,00
(0), (1- (oh,0) pb,k (0 (10)
explsh.
In this formulation, p/,e- E. exp(=5,1/7) represents the assignment probability of the 8th instance's
statistics in the batch belonging to the hthh statistics component. Note that, Pb,, if 6th instance exhibits a greater affinity to the 4th component, and vice-versa. We employ the refined feature statistics (ni, σ²) normalize z', yielding normalized features 2' elaborated upon below:
2/.e. Pb. the0
c... = 7a
Se,
(11)
In the concluding steps, all BN statistics components undergo refinement. This is achieved by updating them based on the weighted average difference between the current instance statistics, with weights drawn from the corresponding assignment probabilities across the batch:
Mhec EErh ph.u (Ph.o phi,)::
A
Pk.c
(12)
(o +- (oh.c)² A B
Ph. : ((ob,o) 2 (ok
(13)
where À is the momentum hyperparameter and is set based on the principles of momentum batch
normalization as proposed by (Yong et al. 2020) (see Appendix A.4)) The more statistic component is closely aligned with the instance- wise statistics in the test batch (precisely when Pb, ~ 1), the more it undergoes a substantial update.
3 EXPERIMENTS
3.1 EXPERIMENTAL SETUP
To maintain faimess in comparison with baselines, experiments carried out using the open- source online TTA repository (Marsden et al., (024). which amasses source codes and configurations of state-of-the-art TTA methods. Implementation details are elaborated in Appendix B.1
Datasets and models. To examine the repercussions of common corruption. we use the Robust- Bench benchmark (Hendrycks & Dietterich) 2019). This benchmark encapsulates the CIFAR1O-C. CIFAR100-C, and large-scale ImageNet-C datasets, offering comprehensive view of 15 diverse corruption types, each implemented at five distinct severity levels applied to the corresponding clean test datasets. Our chosen backbone models consist of WideResNet-28 (Zagoruyko & Komodakis 2016), ResNeXt-29 (Xie et al., (017). and ResNet-50 (He et al.) CIFAR100, and ImageNet, respectively. To assess the robustness of our method natural domain shifts, we utilize the subset of the DomainNet dataset (Peng et al. 2019), renowned for its pronounced domain shifts, focusing on classification tasks. Given the presence of noisy labels in the original
2016). each trained on CIFARIO,
5

---

Published as a conference paper at ICLR 2024
DomainNet dataset, we adopt the approach from (Chen al. 2022) and utilize a refined subset, DomainNet- 126 (Peng et al. 2019), which features 126 classes across four distinct domains: Real, Sketch, Clipart, and Painting. For every domain, a single ResNet-101 model is trained following the (Chen et al., 2022) and, subsequently, evaluated against the remaining three domains. Furthermore, to evaluate our model's resilience in non-i.i.d. video frame-wite classification contexts, we introduce corrupted versions of ImageNet-VDD VID (Russakovsky et al. 2015) and LaSOT (Dave et al. 2020), named ImageNet-VID- VID-C and LaSOT-C, respectively. Each dataset modified with three types of corrpttions: Gaussian noise, artificial snow, and rain, to challenge the models further. As backbone, we use ResNet-50 pre-trained on an original, uncorrupted training set of ImageNet.
Baselines. In our evaluation, we have benchmarked UnMix-TNS -NNS against a diverse range of other test- time normalization methods. Our comparative analysis includes test-time BN recalibration approach (TBN) (Nado et al., 2020), a-BN (You et al. (021). the Instance Aware BN (IABN) layer introduced in NOTE (Gong et al.J 2022), and the Robust BN (RBN) layer proposed in ROTTA (Yuan et al., 2023). Beyond this, we investigate the advantages of pairing UnMix-TNS with different test-time optimization methods, comparing it to standard BN layer usage. In our assessment, we explore several TTA methods, including TENT /Wang et al., 2020), which leverages entropy minimization to fine-tune BN affine parameters, and CoTTA (Wang et al. (022), which optimally employs the Mean teacher method. Additionally, we examine the parameter-free strategy, LAME (Boudiaf et al. 2022), which adjusts model outputs based on batch predictions. As for TTA methods utilizing memory banks to simulate i.i.d. samples, we assess NOTE (Gong et al. (202)) and RoTTA (Yuan et al. 2023).
We also explore the more recent universal TTA method, ROID (Marsden et al. 2024), incorporating various regularization techniques, including diversity weighting.
Evaluation protocols. All experiments are conducted in an online non-i.i.d TTA setting, with immediate evaluations of predictions. Following the non-i.i.d protocols outlined in (Marsden et al. 2024), we first explore the single domain adaptation scenario, wherein the model sequentially adapts to each domain, with a reset weights upon switching domains. Next, we examine the continual domain adaptation scenario, allowing the model to adapt sequentially across all domains without weight reset. Lastly, we assess the mixed domain adaptation scenario, evaluating performance with test batches composed of examples from multiple domains. This approach enables a concise yet comprehensive analysis of model adaptability in diverse domains.
3.2 RESULTS
Tables [ andd] 2 present the average online classification error rates for our method and baselines on corruption and natural shift benchmarks, following three outlined evaluation protocols. Key observations include:
Table 1: Non-i.idd test-time adaptation on corruption benchmarks. Averaged online classification error rate (in %) across 15 corruptions at severity level 5 on CIFAR10-C, CIFAR 100-C, and ImageNet- C, comparing single, continual, and mixed domain adaptation. Averaged over three runs.
Taaao Sounce
CHRABING
CITARIONC Imaghhin
Mon S7.3
CITARIOG
CIPARIOS-C
Maan
CIRARIBC
CEASDOOC Inseliin
Mac
433
46.5
80.0
45.5
46.5
$2.5
$7.5
433
46,5
$2.
31.3
Donn-Tis NISEWNRLAATTO
31.5
M0.5
94.1
96.6
JABN
29.1
55.7
35.0
29.1
56.6
55.7
550
REN
5433
44.5
71.3
$4.7
21.3
$2.0
77.3
92.4
91.1
$3.5
DaMis-TNS
27.8
39.2
78.6
45.6
2688
30.2
76.4
45.5
41.9
38.1
64.3
56.5
TEST- TIME ADAPPEATION
760
(UMMIs-TNS)
50.8
COTTA
71.2
$0.9
82.7
$0.3
77.8
$1.2
42.5
50.5
81.4
94.3
06.7
90.5
70.7
07355
46.6/3335
40.8 (30.5)
21147117
88 (0000
72. 1016800
66.60003
856/1111
76.8
(116
106
24.1
10.6
76.3
48.4
3.8
65.0
(UUMIL-TNS))
5410522
33.7
30.6
63.8
80:
4.2 (004)
NOTE
2500
$3.4
85.0
183303533
453
287 (4000)
58.5 (33.30
7508
48 18.51330
526 12614
51.8033
RoTTA
213
45.5
70.3
22.0
44.5
66.9
42.0
600
65.0
66.7
71.9
260
7004 80
45.7
26.8
38.9 (5.4)
(9.67)8.7)
(300
55.3 0018)
K3
6008
73.4
77.8
17.5
03.5
(TUMMI--TNS)
32.0
UnMix MNNS exemplifies resilience under non-i.i.d. test-time adaptation. Our observation reveals that, in comparison to other test-time normalization based methods, UnMix-TNS stands superior, delivering exceptional average classification performance in both single and continual domain

---

Published as a conference paper at ICLR 2024
Table 2: Non-Li.d. test-time adaptation on natural shift benchmark (DomainNet-126), Online classification error rate (in %) depicted for each source domain, averaged across the other target domains. A comparative analysis between single, continual, and mixed domain adaptation is presented. Averaged over three runs.
SINGLE DOMAIN
CONTINE CONTENISM
DOMALI
MIAKROOMANN
Soare
88.5
41.4
65.3
453
45.4
40.5
+1.6
65.3
45.3
454
40.3
41.6
452
45.3
45:4
81.4
611
$3.5
674
013
-0-4
0111
3533
21.4
5.4
11.
81.3
313
52.4
JABN
55.2
022
9712
15.3
69.2
06-2
4231
67.2
00.3
06.2
88.2
623
8722
03.3
68.2
46.9
423
65.5
41.3
45.5
+8,0
41.6
452
40.6
46.5
46.
41.6
67.6
35.5
44.4
TENE
00.2
$3.
$5.5
$74
81.
16-3
2712
15.0
$1.4
17.5
53.3
531
91.3
023
52.4
CUTrA
88.3
$5.4
$74
55.
66.3
07.7
$2.
87.4
11.5
63.
031
96.
013
92.4
LAME
12.3
2411
25.3
12.5
315
32.2
2011
20.0
523
30.
21.4
012
11.
170
NOTE
21.3
07.4
991
34.3
15.0
0026
56.6
36-1
54.4
57A
23.5
002
53.3
37.8
AoTTa
21.3
35.4
15.
55.1
36.7
33.4
461
52A
65.2
6-2
00.2
916
ROAD
11.4
55.
651
55.6
18.4
86.
15.1
02.2
66.
63.2
033
933
Table 3: Non-i..d. test-time adaptation on corrupted video datasets. We adapt the ResNet-50 backbone trained on ImageNet on the sequential frames of ImageNet-VID-C and LaSOT-C and report classification error rates (%). t denotes methods using UnMix- -NNS layers.
Dutaset
Coruption
Source
UaMIs-TNS
TENT
LAME
LAME'
Gauss, Noisc
79.6
92.4
78.1
74.1
92.4
76.3 (-16.)
78.0
71.2 (-6.8
VID-C
Rain
$2.4
92.4
83,0
80.0
92.4
80.8 (-1.6))
80.4
76,8 (-3.6)
Snow
49,2
91.4
61.9
91.4
(.366)
45.7
46.4 (40.7)
Moan
70.4
92.0
74.4
68.6
92.0
7-.61-21.41
68.0
64.8 (-3.2)
Gauss, Noise
84.3
97.2
56.7
82.9
97.2
82.7
82.6
80.2 (-2.4)
LaSOrCC
Rain
89.5
97.1
89.5
88.6
97.1
(-8.7)
87.5
87.3
Snow
71.2
96.1
784
71.9
96.1
72.1 (-24.0)
66.5
66.8 (40.3)
Mean
81,7
96.8
84.8
81.1
96.8
61.0
78.9
7K.1 (-0.5)
adaptation, spanning corruption and natural shift benchmarks. More explicitly, method shows an increase in accuracy by 11 11.0% and 11.1% 11 averaged over three datasets on the corruption benchmark, surpassing the second-best baseline. The merit accentuated in DomainNet- 126; the improvement rises to 11 1.55% and 12.5%, corresponding to scenarios where the domain discrepancy is notably profound. In realms of mixed domain adaptation, UnMix-TNS outperforms its closest competitor by 5.6% and 2.6% on CIFAR DIFARIO0-CAAA and ImageNet-C, respectively. This is supported by a remarkable reduction in the error rate by 17.2%, surpassing the next best method on mixed domain adaptation on Domain 126. The insights derived from our experiments are particularly enlightening for non-i.i.d scenarios, suggesting that the update of multiple distinct statistics components achieves superiority over the continuous adaptation of singular component as in RBN or blending source statistics with the incoming batch/instance (a-BN/IABN) target statistics.
Elevating TTA methods with UnMix-TNS. In Tables D and 2 we benchmark against the fore- front TTA strategies, both standalone and when integrated with UnMix-TNS. We note significant performance reductions for TENT, CoTTA, and ROID when operated solely across corruption and natural shift benchmarks, observing error rates escalating to a minimum of 73.4% across every dataset and evaluation scenario. We posit that the reliance of the methods on TBN explains the substantial drops in performance when exposed to non-i.i.d. testing. However, when combined with UnMix-TNS, these methods experience remarkable performance improvement. Notably, ROID achieves gains of at least 45.8% on corruption datasets and 60.1% on DomainNet- domains, ultimately achieving the best mean results in both single and continual domain adaptation for both benchmarks. Conversely, methods like NOTE and RoTTA use memory banks to simulate i.ikd.llike conditions, improving test-time accuracy in single and continual domain adaptation scenarios only on corruption benchmarks. Regardless. they face challenges when dealing with mixed domain or larger domain shifts, as exemplified by DomainNet-126. Instead, we demonstrate that the integration of UnMix-TNS consistently elevates their performance, highlighting the advanced efficacy of our normalization layers compared to the IABN and RBN employed by these methods. Among the TTA methods, only LAME, which propagates labels within batch, exhibits remarkably low error rates in both test scenarios. Nonetheless, UnMix- TNS advances its overall performance further, except in the case of mixed domain adaptation scenarios for CIFAR 100C and ImageNet-C.
7

---

Published as a conference paper at ICLR 2024
Evaluating UnMix-TNS for corrupted video datasets. In Table model, initially trained on ImageNet, adapts at test time using corrupted versions of ImageNet- VID and LaSOT video datasets. UnMix-TNS consistently surpasses the test-time normalization schemes of TBN by 23.4/15.7% and e-BN by 5.8/3.7% on ImageNet- VID/LaSO// respectively. It also demonstrates overall improvemett of 21.4/15.8% and 3.2/0.8% for TENT and LAME, respectively. Moreover, UnMix-TNV proves to be more robust than other test-time normalization schemes, predominantly improving upon the baseline source accuracy, especially in scenarios affected by covariate-shifee video frames, where others may experience a noticeable decline in source model performance.
assess how ResNet-50
3.3 ABLATION STUDIES
Deciphering test sample correlation impact. This study, aligned with (Yuan et al. 2023), adopts the Dirichlet distribution, synthesizing correlatively sampled test streams through the concentration parameter 5 to investigate their impact on domain adaptation under continual, and mixed domain adaptation. A decrease in results in heightened correlation among test samples and category aggregation, depicted by different 6 values in Figure (a) on the CIFAR RO-CC dataset. Concurrently, this decrease triggers a pronounced decline in the performance of TBN, a-BN, and RBN, owing to their indifference to the rising correlation among test samples. Notably. UnMix-TNS exhibits more resilience regarding various values of 8, demonstrating its effectiveness in the above-mentioned non-i.i.d. test-time scenarios. Further ablation studies concerning single domain adaptation, along with experiments on CIFARIO-C, are provided in Appendix
Effect of batch size. While our experiments are primarily conducted with a batch size of 64, we also explore the effect of varying batch sizes, from to 256, on temporally correlated streams within continual and mixed domain adaptation to address potential curiosities regarding batch size influence. As illustrated in Figure 3 (b), we observe consistent decrement in the error rate for TBN and RBN as batch size increases, while a-BN's error rate first increases with batch size and then reduces afterward. This occurrence reflects the premise that an increased batch size facilitates the incorporation of more categories within each batch, thereby diminishing the label correlation therein. For a-BN, a batch size of 1 corresponds to the computation of normalization statisties on an instance-wise basis, utilizing a blend of stored and current statistics, a method which, intriguingly, yields the best performance. Distinctly, UnMix-TWe -TNS remains robust to batch size variations, consistently delivering superior results across adaptation scenarios over a wide range of batch sizes. Additional results for single domain adaptation, along with experiments on CIFARIO-C, are provided in Appendix C
UnMix- TAMix-TNSAA introduces minimal computational overhead. To accurately assess the efficiency of the UnMix-TNS, we execute precise computations of the inference time over the 15 corruptions of CIFAR10-C. These calculations are uniformly performed under consistent running environ- ments meets--utilizing an NVIDIA GeForce RTX 3080 GPU and maintaining the same batch size of 64 and K=16. Despite the negligible additional inference time cost-a mere extra 0.15ms per image compared to vanilla source inference the integration of the proposed UnMix-TNS results in a substantial enhancement of 16.5% in average accuracy. When integrated into a method like TENT, the inference rate slightly increases by 0.58ms per image for a significant average accuracy gain of 49.0%
In Appendix C we provide supplementary ablation studies and experiments, focusing on the effect of the depth of UnMix-TNS layers in neural networks and the number of components, denoted K.
4 RELATED WORK
Online Test- Time Adaptation. Prominent TTA methods, such as self-supervised tuning (Sun et al., 2020) Liu et al.) (202T). batch normalization recalibration (Nado et al.) 2020 Wang et al. 2020 and test-time data augmentation (Chen et al., 2022 Tomar et al. (0023), are often limited to specific experimental setups. These methods assume single stationary distribution shift, large batch sizes, and consistent labels within test batches, leading to suboptimal performance in diverse testing scenarios. Hence, recent efforts have focused extending to more practical testing scenarios. For instance, CoTTA (Wang et al. 2002) focuses on adapting to evolving target environments but relies on i.i.d. test data. They employ dual utilization of weight and wwwpeenttttio-vvvrggeee predictions,
8

---

Published as a conference paper at ICLR 2024
Source
7BM
Q-38
RBN
UnMix TMA
CONTIRUAL DONATM
NIXED DOMATK
CONTINUAL DOMATN
DONATN
100
foroe TAtA 00
80
60
ualforn
10.0
1.0
0.1
0.01
uniferm 10.0
1.0
0.1
0.01
2
15 52 54 125255
15 52
54
126:255
(a)
6
(b) Batch 8i2e
Figure 3: Ablation study on the impact of (a) Dirichlet parameter, 8, and (b) batch size on CIFARI00- C, comparing several test-time normalization methods including TBN, a-BN, RBN, and UnMix-TNS.
resulting in substantial model updates and computational overhead. LAME (Boudiaf et al. 2022 suggests non-i.i.d. test-time adaptation based on batch predictions, but it may be sensitive to batch size fluctuations. NOTE (Gong et al. 2022) and RoTTA (Yuan al. 2023) use memory bank for category- balanced data, effective under non-i..d. and non-stationary contexts but with high memory requirements. ROID (Marsden et al. 2024) introduces universal test-time adaptation with various protocols, benefiting from strategies like diversity weighting and normalization layers like group or layer normalization for improved resilience to correlated data. However, its effectiveness diminishes in non-i.i.d. scenarios with BN layers in the backbone.
Normalization in Test Time. Test-time BN adaptation methods have emerged that utilize test batch statistics for standardization (Nado et al., (220) or blending source and test batch statistics (Schneider et al. (2200) to counteract the intermediate covariate shift adeptly. Similarly, methods like a-BN (You et al. 2021) and AugBN (Khurana et al. 2021) integrate both statistics through the use of predetermined hyperparameters. Other methods modify statistics via a moving average while augmenting the input to create virtual test batch (Hu et al.) 2021 Mirza et al., (022). For instance, the MixNorm (Hu et al.) (02T) uses training statistics as global statistics, updated through an exponential moving average of online test samples, even for single sample. InstCal (Zou et al.)
2022) introduces an instance-specific BN calibration for test-time adaptation, bypassing extensive test-time parameter fine-tuning. NOTE (Gong et al. (0022) has presented instance-aware BN (IABN) to correct normalization of out-of-distrion oisriitribbttinn samples. In another concurrent work, RoTTA (Yuan al. 2023) suggests robust BN (RBN), which estimates global statisties via exponential moving average. Recently, TTN (Lim et al., (223) introduced test-time normalization layer that merges source and test batch statistics, leveraging interpolating channel-wise weights to seamlessly adapt to new target domains while accounting for domain shift sensitivity. However, it is essential to highlight that TTN relies on prior knowledge of the source data, representing a slight departure from traditional test-time adaptation methods.
5 CONCLUSION
This paper proposes UnMix-TNS, a novel test-time normalization layer meticulously designed to counteract the label temporal correlation, particularly in the context of non-i.i.d. distributionally shifted streaming test batches. UnMix-TNS, inherently versatile, integrates seamlessly with existing test-time adaptation techniques and BN-equipped architectures. Through rigorous empirical testing on various benchmarks-innlluinn both corruption and natural shift benchmarks of classification, as well as newly introduced corrupted real-world video datasets, we provide compelling evidence of robustness across varied test-time adaptation protocols and the significant performance enhancements achievable by leading TTA methods when paired with UnMix-TNC in non-i.i.d. environments.
Limitations. Future work will concentrate on two main areas: firstly, adapting to scenarios where test batches include wide range of diverse outlier instances; secondly, applying our method to BN-based segmentation models during test-time adaptation to enhance its adaptability. Additionally, determining the ideal number of inMix-TNS components will be explored further, as different datasets and adaptation scenarios may require varied optimal values.

---

Published as a conference paper at ICLR 2024
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1007.0660.066 2016.
Philipp Benz, Chaoning Zhang, Adil Karjauv, and In So Kweon. Revisiting batch normalization for improving corruption robustness. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 494-503, 2021.
Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-free online test-time adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern 2022.
Recognition, pp.
Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 295-305, 2022.
Lechao Cheng, Chaowei Fang, Dingwen Zhang, Guanbin Li, and Gang Huang, Compound batch normalization for long-tailed image classification. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 1925-1934, 2022.
Achal Dave, Tarasha Khurana, Pavel Tokmakov, Cordelia Schmid, and Deva Ramanan. Tao: large-scale benchmark for tracking any object. In Computer Rision-ECCy 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part 16. pp. 436 454. Springer, 2020.
Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. Note: Robust continual test-time adaptation against temporal correlation. Advances in Neural Information Processing Systems, 35:27253-27266, 2022.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: 1903. 12261, 2019.
Xuefeng Hu, Gokhan Uzunbas, Sirius Chen, Rui Wang, Ashish Shah, Ram Nevatia, and Ser-Nam Lim. Mixnorm: Test-time adaptation through online normalization estimation. arXiv preprint arXiv:2110.11478, 2021.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. pmlr, 2015.
Mahdi M Kalayeh and Mubarak Shah. Training faster by separating modes of variation in batch- normalized models. IEEE transactions on pattern analysis and machine intelligence, 42(6): (480-1500, 2019.
Ansh Khurana, Sujoy Paul, Piyush Rai, Soma Biswas, and Gaurav Aggarwal. Sita: Single image test-time adaptation. arXiv preprini 12Xiv:2112.023. 02355. 2021.
Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi. Tin: A domain-shift aware batch normalization in test-time adaptation. arXiv preprini arXiv:2302.05155. 2023.
Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. Ttt++: When does self-supervised test-time training fail thrive? Advances in Neural Information Processing Systems, 34:21808 14421888-228200 21820, 2021.
Robert A Marsden, Mario Döbler, and Bin Yang. Universal test-time adaptation through weight ensembling, diversity weighting, and prior correction. In Proceedings of the IEEE/CVF Winter Conference Applications Computer Vision, pp. 2555-2566 2024.
10

---

Published as a conference paper at ICLR 2024
M Jehanzeb Mirza, Jakub Micorek, Horst Possegger, and Horst Bischof. The norm must go Dynamic unsupervised domain adaptation by normalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14765-14775, 2022.
Zachary Nado, Shreyas Padhy, D Sculley, Alexander D' Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963, 2020.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In Internetional Conference on Learning Representations, 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-perfore sesh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1406-144 1415, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252. 2015.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in neural information processing systems, 33: 53:113: 1539-11551, 2020.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-suuervision for generalization under distribution shifts. In International conference on machine learning, pp. 9229-9248. PMLR, 2020.
Devavrat Tomar, Guillaume Vray Behzad Bozorgtabar, and Lean-Philippe Thiran. Tesla: Test- time self-learning with automatic adversarial augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20341-20350, 2023.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprinl arXiv: 1607::607..0002 2016.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test- time adaptation by entropy minimization. International Conference on Learning Representations, 2020.
Qin Wang. Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7201-7211, 2022.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference computer vision (ECCV), pp. 3-19, 2018
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 1492-1500,, 2017.
Hongwei Yong, Jianqiang Huang, Deyu Meng, Xiansheng Hua, and Lei Zhang. Momentum batch normalization for deep learning with small batch size. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16, pp. 224-240. Springer, 2020.
Fuming You, Jingjing Li, and Zhou Zhao. Test-time batch statistics calibration for covariate shift. arXiv preprint arXiv:21 10.04065, 2021.
11

---

Published as a conference paper at ICLR 2024
Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15922-15932, 2023.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
Yuliang Zou, Zizhao Zhang. Chun-Liang Li, Han Zhang, Tomas Pfister, and Jia-Bin Huang. Learning instance-specifie adaptation for cross-domain segmentation. European Conference on Computer Vision, pp. 459-476. Springer, 2022.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX: PROPERTIES OF UNMIX-TNS COMPONENTS
This section provides supplemental material for Sections 2.2. and 2.2.2
A.1 STATISTICS OF K COMPONENT MIXTURE DISTRIBUTION
Let fx(x) = Ee wi (x) be the probability distribution of a random variable X R. which is a linear sum of K distinet probability distributions, such that wi 2 0 and E3 W/ = 1. Also, let Ni = fxx 1 (zx) dx, and of =f mx)² f& (z) dx. The mean and variance o² of random variable X under the distribution fx (1r), are computed as:
n = 1 (x(x)) dx = f x. EEn wk (x(x)) dx Σ
wk f x. Sx (x) dr Σ Wklfes (14)
k
= (x- -x(x))x(x))))))))))))))))))))))) dx / (x-10² (E wk 18 (x)) dx
0² =
Σ Wk (fz2 1& (x) de 122 f (x) de - 2/1 f x. 18x dx
k
Σ wh (0 + (x) + 11² 2p MK)
k
oe + Σ 1 Σ k
wk+
k
k
A.2 INITIALIZING INDIVIDUAL STATISTICS OF K UNMIX- TNS COMPONENTS
Let us initialize the means and standard deviations (nf: of) time / of the individual components of the UnMix-TNS layer, such that, in expectation, the statistics of their mixture distribution are equal the stored statistics (H, o)) the corresponding BN layer. For simplification in notation, we drop at O and arbitrary channel e as
and c, and represent the mixture distribution from Section 2.2. follows:
1 Σ N(K,. oK) K
hz(2)=
(15)
where we define μk 1u + Çt and dg = where GA NN( c). Then, by applying Equations 14 and (15.. the expected value (n, 22) of the mean and variance of mixture distribution can be expressed as follows:
Ela
I K
E(u (u
(w) (K) = n,
(16)
Σ (H E(m+Ga)) Ge) 2)
E
E
p²
1
Σ (# +xa=2²=
1
K
K²
p² (1 K 2
(17)
Note that the right-hand side of the above equation remains constant. Consequently, at initialization, creating more diverse components (high e) necessitates lower individual variance of the components, p². Thus, the maximum value of 2 is established as 2 K-1 K J2 and minimum value as
2
Hence, we proportionally scale between (0, VO.VKK
utilizing the hyperparameter o, K
i.. e
a.K K-I
13

---

Published as a conference paper at ICLR 2024
A.3 ADAPTING UNMIX- TNS TO TEMPORALLY CORRELATED TEST DATA: A THEORETICAL PERSPECTIVE
Let h(z) be the true distribution of the test domain features. We assume that l1z)) can be decomposed as a mixture of K Gaussian distributions [=)] such that k(z)) = K Σx (z). Additionally, we postulate the existence of a perfect classifier F Z - (Y, K)) that deterministically assigns each feature 2 to its corresponding label y and component k. expressed as (y. k) = F()).
In the context of independent and identically distributed (i.i.d.) features, each z uniformly sampled over time with respect to its corresponding label y (unknown to the learner), and thus the expected value of the mean (utilized for normalization) of the current batch can be calculated as follows:
_ h(2,8,k)dz = EJ h(zlk)h(ylz, k))(k)) W.k
E Hbasch] = p2 =
(18)
v.k
where h(2, y. k) represents the joint distribution of the features 2, labels and the component k. The term h(2[k) = (2) denotes the probability distribution of the features given a particular component k. while l(kk- = K implies that each component is equally likely. Furthermore, 1(yl=,, k)) is the conditional probability distribution of the labels given the features and the component Since the perfect classifier F allows for the deterministic determination of y, k from it follows that Σy (yy|,, k) 1. This can be rearticulated in the above equation for estimating the expected mean
μ²" as follows:
1 -²=R>>
E 2- h2(2))((222) k)dz = X E U.k
1 K k
hx()d
Σ n R
(19)
K
where 1 is the mean of the k component of the true distribution of the test domain features.
In a non-i.i.d scenario, we sample the features 2 to ensure a temporal correlation with their corresponding labels y. As a result, the true test domain distribution l(z)) given time is
approximated as (2(2)) = Ex (x), where the distribution of the wth component hx(2) is
approximated (x). This leads to the estimation of an unbiased normalization mean time t using (z). as expressed in the following equation:
{
1 K
Σ 2-
MUAME-TNS (t)
(20)
Furthermore, the bias SOnMix--NS(( (t) between the mean of the true test-domain distribution μ² and the estimated mean PUaMix-TSS (f) time & can be recorded as:
1 UnMix UnMix-TNS(( TNS(1) = K E 2. (ha(z) -
1000----cen Mz = K
-UG 1 -
(21)
where üig is the estimated mean of the same ttth component at time t. Note that time progresses and t increases, fif. - ngg as we update jit using exponential moving average mechanism (Equations 12 & 13 in the paper). Thus, UnMix-TNS can help mitigate the bias introduced in the estimation of feature normalization statistics, which arises due to the time-correlated feature distribution. Conversely, the expected value of the normalization mean estimated using the current batch (TBN) for the non-i.i.d. scenario can be defined as:
Pran(t)= = E
=X
h' h2
=
E h(2)k))())) k)h (k)) 2 - hu(2))
y.k
B.kc
where 1(((k) represents non-i.i.d. characteristics of the test data stream. In this case, the bias is obtained as follows:
ATox(t) = =1² - ((((K)
(22)
This equation indicates that the test samples are uniformly distributed over time (i.e., in an i.i.d. manner), where h' (k) 1/K, the estimation of normalization statistics will not be biased. However, in situations where h (k) smoothly varies with time, favoring the selection of a few components over others, TBN will introduce non-zero bias in the estimation.
14

---

Published as a conference paper at ICLR 2024
A.4 OPTIMIZING MOMENTUM: INSIGHTS FROM MOMENTUM BATCH NORMALIZATION
Our approach to setting the optimal value of the hyperparameter À for momentum draws inspiration from the concept of Momentum BN (MBN) (Yong et al. 2020). The effectiveness of BN has traditionally been attributed to its ability to reduce internal covariate shifts. However, MBN (Yong) ct al., 2020) has provided an additional perspective by demonstrating that BN layers inherently introduce a certain level of noise in the sample mean and variance, acting as a regularization mechanism during the training phase. key insight from MBN is the relationship between the amount of noise and the batch size, and smaller batch sizes introduce relatively larger noise, leading to less stable training process. The rationale behind MBN is to standardize this noise level across different batch sizes, particularly making the noise level with small batch size comparable to that with a larger batch size by the end of the training stage. To achieve this, MBN modifies the standard BN approach: instead of directly using the batch means and variances in the BN layer, MBN utilizes their momentum equivalents as follows:
1+1 = MC
= (1- A)pe + AuB = 12 + M(BB 122)
(23)
=1 -1- + Mon))² = (o')² + A(((n)) - (o223)
(24)
Here, t denotes the thh iteration, n²a and of represent historical means and variances, μ4 and d represent the current batch means and variances, and is the momentum hyperparameter, MBN introduces additive noise E,. ~ NO0 à and multiplicative noise & with a Generalizedd-Ci-ssquarre B
2(-11) These formulas show that distribution with expectation E(c,] B and Var(S,] = -ーx B²
B
smaller batch sizes lead increased noise but also reveal that the noise level can be moderated by the momentum hyperparameter A. Based on this insight, MBN proposed a formula to determine a robust given the batch size B:
A A=1- (1-))) B/Bo B//B
(25)
where Ao and Bo represent the ideal momentum parameter and ideal batch size, respectively. In- tuitively, a smaller batch size leads to a lower X, thereby reducing noise generation. Given that Equations (12) and (13) in our method are similar to those in the MBN approach Equations (23)
and (24). we posit that our method might also encounter significant noise with small batch sizes. To
address this and ensure stability across varying batch sizes, we adopt the hyperparameter following
Equation 25 using Bg- = 64 and An = 0.1, aligning with (Yong et al. 2020).
B APPENDIX: FOR REPRODUCIBILITY
B.1 IMPLEMENTATION DETAILS
All experiments were performed using PyTorch 1.13 (Paszke et al. 2019) NVIDIA GeForce RTX 3080 GPU. For the CIFAR 10-C and CIFAR100-C, we optimize the model parameters of the test-time adaptation methods utilizing both BN and UnMix- layers with the Adam optimizer with a learning rate of le-5, no weight decay, and a batch size of 64. For the DomainNet-126, ImageNet-C, ImageNet- -IDC-C and LaSOT-C datasets, we use the SGD optimizer with a learning rate of 2.5e-6, momentum of 0.9, and no weight decay, with a batch size of 64 for DomainNet- 126 and 16 for the remaining datasets.
In implementing our method, we set to 0.5 in all our experiments. As for the number of UnMix-TNS components K, we set for single and continual test-time adaptation on CIFARIÓ-C, CIFAR100-C. NomainNet126CC. and ImageNet-C, while setting K to 128 for mixed domain test-time adaptation. For mixed domain adaptation, K is increased to 128 to aptly represent a diversity of domain features within the neural network and is further adjusted to 256 for ImageNet- VID-C and LaSOT-C. This higher K value is pertinent to accommodate the heterogeneous domain features inherent in mixed domain adaptation.
The 8 parameter controlling non-i.i.d. shift of the Dirichlet sampling distribution is set to 0.1 for CIFAR 10-C and adjusted to 0.01 for CIFAR100-C, 100-C, ImageNet-C, and DomainNet- 126.
15

---

Published as a conference paper at ICLR 2024
B.2 PSEUDOCODE
We provide PyTorch-friendly (Paszke et al. 2019) pseudocode for the implementation of the UnMix- TNS layer, referenced as Algorithm
Algorithm 1: PyTerch-friendly pseudocode for the UnMix T layer
def init_ 8, source_mean source.var momentum, X)
ini-islizetion
self.7 7, self. $ choose a 0.5, C is number of charne 3 source_mean.
self nementum
ini=ialize K random UnMix-The components
torch.
(K-1)) torch.randa (8, C) shape: (K,C)
self. corponent
bersortensor ([nciseli] source.mean for in range (3)
shape: ()
self. wowponent.varr. torch. torch.tansor (11-a)
for in (X)
shape: ()
(x)
has shope: (3,
instance.mean, irstance.val = torch.var.meantg dim=(2, 311 shapes (8, compute assigament probabilities
with torch.
no gradients
(im mear, self. www.pccnttttmmman.a.....
shopa: (8,
aoftmax isim / 0.97, din-l) uunsuuezee (-1) à shape: 18, 1,18,
mix the instance statistics with. components" statistics
(1-p)+self componnot shaper (B,
unaqueeze peirstance (nsquee7e
hat.var (l-p) ctteel..coooonntt.arrr.. unsqueeze
shape: (2,
compute instange-wisn normal
tormalization ization
torch.mean (hat_mean, dim-l1
torch.meen (hat.var, d_m=1) torch.meen (hat mean""2 dim-l) 4
update component's statistion
with torch. a)ugzad():
no gradlents
update componenta'
self. component ierans means = self colf.commonent.menne
molffmomenttum
itorch, mean (hat meaa, dim=()
sall.coommmentteeee meanss)
update X composental vazs
self. componett vars self, component vars self momeettm (torch. mean (hat var, dim=0) colpoeenp))))
normelize festures
K_norm μ) torch. (o² return x.norm self.y self,8
C APPENDIX: ABLATION STUDIES AND ADDITIONAL EXPERIMENTAL RESULTS
C.I EXPLORING UNMIX-TNS INFLUENCE ACROSS THE VARIED DEPTHS OF NEURAL NETWORK LAYERS
Figure (a)-(b) illustrate the effect of replacing the BN layer with UnMix-TNS layers in the early or deeper segment of the neural network, respectively. The incorporation of UnMix-TNS proves to be essential for the early and intermediary layers of the neural network for combating the challenges posed by non-i.i.d. test streams. However, the deeper layers, employing the BN layer synchronized with initial source statistics is discemed to yield superior results.
C.2 HYPERPARAMETERS SENSITIVITY
In this section, we present supplementary results of our hyperparameter sensitivity analysis, which serves complement analysis of the impact of test sample correlation and batch size the performance test-time normalization-base methods, UnMix-TNS included. Furthermore, we delve into the sensitivity of UnMix- concerning the number of its components K.
Impact of sample correlation and batch size. normalization layers, in response variations in the concentration parameter å and batch size, is
The resilience of UnMix-TNS, along with other
16

---

Published as a conference paper at ICLR 2024
42.5
42.5
40.0
40.0
Oeror rate (3)
37.5
36.0
Brror rate
36.0
82.6
30.0
27.5
27.5
layor index
layer index
(a)
(b)
Figure 4: Exploration of UnMix-TNS influence at varied depths within the neural network. (a) Represents the average classification error rate when only the BN layers subsequent to the layer index are replaced by UnMix-TNS layers. (b) Shows the average classification error rate when solely the BN layers preceding the layer index are exchanged by UnMix-TNS layers. A layer index of corresponds to the first layer. The depicted experiments focus on non-i..d. continual test-time domain adaptation the CIFARIO-C.
depicted in Figure Furthermore, in Figure 6 provide additional insights into sensitivity analysis for CIFARTO-C and CIFAR100-C showcasing results for single test-time domain adaptation. In summary, these additional observations corroborate the analysis in main paper, underscoring that UnMix-TNS maintains remarkable stability against alterations in batch size and 8, consistently surpassing the average performance of baseline methods.
Influence of the Number of UNMIX-TNS components K. In Figures 7 and influence of the hyperparameter K on UnMix-TNS when used both independently and in conjunction with ffaee-oftthe-ar TTA methods. Our results indicate that for both single domain and continual domain adaptation, K = 16 yields the best performance. However, increasing the value of K results in higher error rates on CIFARIO-C, while the performance remains stable on CIFAR100-C, This discrepancy can presumably be attributed to the limited class range in CIFAR10-C, implicating lesser feature diversification. On the other hand, when comes to mixed domain adaptation, our findings demonstrate that higher value of K is beneficial for UnMix-TNS. Mixed domain adaptation scenarios typically involve more class-wise feature diversity within batch, which can be effectively handled by increasing the value of K.
explore the
Source TON O Q-DB
Unllix TMB
CONTINUAL DOXAIN
MIXED DONAIK
CONTINUAL DONA
XIXED DOMAIN
Breer rate
uniforric.a 1.0 3.1. uniforato.co 1.3
0.01
1362566
105 255
(a)
(b) Batch size
Figure 5: Ablation study on the impact of the (a) concentration parameter 8, and (b) batch size on CIFARIO-C for several test-time normalization methods including TBN, a-BN, RBN, and our proposed UnMix-TNS.
C.3 EXPLORATION OF AUGMENTATION TYPES
In the post-training phase, we leverage data augmentation means simulate potential domain shifts, with the primary objective being to subject the model to a spectrum of imput domains similar the corruption benchmark. This simulation is pivotal, allowing an in-depth analysis of the model
17

---

Published as a conference paper at ICLR 2024
Source
7BM
Q-38
RBY
UnMix-TMM
CIFARIOG-C
CIFARICO-C
100
foroe TAte 03
40
uaiford 0.0 0. 1.0
0.1 0. 0.001
uniterm 10.0
1.0 0.1
0.01
2
15 52 54 125255
15 54
196255
(a) 6
(b) Batch 8i2e
Figure 6: Ablation study on the impact of the (a) concentration parameter 8, and (b) batch size on CIFAR and CIFAR100-C for single domain adaptation. We compare several test-time normalization methods, including TBN, a-BN, RBN, and our proposed UnMix-TNS.
TENTY
CoTTAT
LAMET +B- NOTET + RoTTAT - ROIDÝ UnNix-TNS
rata 00
60
64 k
1024
1024
k
Figure 7: Ablation study on the impact of the number of UnMix-TNS components K on CIFARIO-C.
The symbol f denotes indicates the employment of LaMix-TNS in the method.
influenced by any discrepancies between shifted and original domains. It is crucial to understand that the essence of this examination is not in the final domain to which the input is altered but in the alteration the domain itself. This is demonstrated through the ablation study of augmentation types applied CIFAR10 10 and CIFAR 100 using pre- pre-trained WideResNet-29 and ResNeXt-29 in Table
For this ablation study, color jittering is employed as the base augmentation, and we sequentially incorporated random grayscale, Gaussian blur, and random horizontal flip-each augmentative step reflective, though not identically aligned, with the corruption types present in our corruption benchmark. Our observations reveal that UnMix-TNS consistently maintains lower average error rate across all augmentations, aligning closely with the source model and even surpassing it in instances such as Gaussian blur. These findings are supported by corresponding non-i.i.d. test-time adaptations on corruption benchmarks, detailed in Table substantiating the versatile efficacy of UnMix-TNS in navigating diverse domain alterations while preserving the accuracy of the model.
C.4 UNMIX- SAFEGUARDS THE KNOWLEDGE SOURCED FROM THE ORIGINAL. DOMAIN
In practical scenarios, data originating from the source domain may resurface during test time. For this purpose, we employed clean domain test data from CIFAR10 and CIFAR100 datasets in single domain non-i.i.d. adaptation scenario to demonstrate how UnMix-TNS and other test-time normal- ization methods acclimate to previously encountered source domain data, albeit unseen instances. As depicted in Table 5, all baseline strategies employing test-time normalization layers exhibit a decrement in performance, regardless of the expansion in batch sizes. This leads to infer that relying on source statistics amassed from extensive training data is still preferable to depending solely current input statisties. However, UnMix-TNS distinguishes itself by continuously refining the
18

---

Published as a conference paper at ICLR 2024
TENTY
CoTTAT
LAMET
NOTET
BoTTAT
ROIDT
SANGLE DOMAIR
CONTINUAL DOMALN
MIXED DCMAI9
80
Error (1)
4
64 296
k 1024
16
64
206
1024
16
256
1924
Figure 8: Ablation study on the impact of the number of UnMix-TNS components K on CIFAR100- C. The symbol i denotes indicates the employment of UnMix-TNS in the method.
Table 4: Robustness data augmentation. The error rate ()) on augmented CIFARIO and CIFAR100 using pre-trained WideResNet-28 and ResNeXt-29, respectively. +X denotes the augmen- tation X is added sequentially color jittering the transformation function.
Dataset
CIFARIO
CIFARI00
Augmentatio
color
*grayscale
+gunssaa blur
+honzonial
color
blur
chorizonial
Source
11.9
55.3
7.1
27.0
53.2
49.5
27.3
TBN
72.0
73.2
76.5
71.8
80.0
84.9
62.4
79.8
e-BN
18.2
23.6
56,0
17.8
40.1
57.9
55.7
40.5
RBN
43.0
47.0
60.7
43.0
38.6
57.5
50.6
39.1
statistics components the normalization layer, initialized from source statistics, whilst concurrently utilizing the current input. This strategy facilitates enhanced preservation of source knowledge, as demonstrated by a more moderate decline performance compared to the source model, particularly on the CIFAR 100 dataset. Moreover, UnMix-TNS manifests stability in non-i.i.d. adaptation scenario and exhibits robustness to variations in batch size. This contrasts with other normalization methods, which necessitate larger batch sizes to produce reliable statisties.
Table 5: Non-i.i.d adaptation in source domain. The error rate (L) on CIFAR 10 and CIFAR100 using WideResNet-28 and ResNeXt-29, respectively.
Method
Source
TBN
n-
a-BN
REN
UnMix-TNN
Batch size
64
64
64
16
16
CIFARTO
15.0)
63.4
8.9
CIFAR100
21.1
77.9
89.0
92.2
34.2
39.3
37.7
32.7
50.3
73.9
25.4
25.6
25.7
Avg.
132
74.4
81.6
84.1
24.6
27.2
25.4
36.7
56,9
72.9
17.0
17.3
17.3
C.5 CORRUPTION-SPERS RESULTS
In Tables to g we provide comprehensive set of results for CIFARIO-C, CIFAR100-C, and ImageNet-C,, focusing single, continual, and mixed domain adaptation. More precisely, delineate error rates for each individual corruption within the benchmark. In the case of continual domain adaptation, the corruptions are ordered based on the test timestamps. These results reinforce our prior analysis from the main paper, offering a more granular examination of the corruption level. In particular, for single domain adaptation, we consistently outperform test-time normalization-hast methods in 14 out of 15 corruption types across all datasets. Within the realm of continual domain adaptation, our approach experiences slightly higher error rates on few corruptions, those of motion, defocus, fog, and contrast with IABN for CIFAR10/100-C. Nonetheless, we sustain superior stance in terms of overall performance. As for ImageNet-C C, UnMix- TNS mirrors the outcomes of RBN in the presence of noisy data but consistently outperforms RBN when subjected to alternative corruptions. These elaborate findings robustly validate the proficiency of UnMix- TNS amidst varying cireumstances and corruption types.
19

---

Published as a conference paper at ICLR 2024
Table 6: Corruption-specifee error rates under three non-i.id. test-time adaptation scenarios. The depicted error rates correspond to the adaptation from CIFARIO to CIFARIO-C on temporally correlated samples by class labels using Dirichlet distribution with 8 0.1 and corruption severity of 5. Methods marked with f denote the integration of our proposed UnMix- TNS with the respective TTA method. Averaged over three runs.
(a) Single domain non-i.idd test-time adaptation.
NETHOB
GAUSS
SHOT
IMPUL
DRFOC.
GLASS
MOTION
Z0OM
SNOW
PROST
FOG
CONTR.
FLASE
PINFL
JPVG
AVG.
Source
72.3
65.7
72.9
47.0
54.3
34.8
42.0
25.1
41.3
26.0
9.3
46.7
26.6
58.5
30.3
43.5
IBN
71.2
73.9
74.3
73.5
72.0
17.3
76.7
76.0
o-BN
60.7
56.3
66.5
41.8
58.0
38.0
40.0
35.4
39.8
31.6
19.2
38.6
11.5
51.0
44.3
11.1
IABN
43.5
40.7
51.2
20.6
-8866
19.5
21.8
22.0
24.0
21.2
11.3
10.9
34.1
30.5
36.6
29.1
RBN
59.8
58.1
65.2
49.2
65.4
50.1
48.7
52.3
51.6
49.4
43.4
488.
57.7
55.2
60.0
54.1
UIMIx-TNS
38,9
35.9
46.6
20.4
41.4
20.1
22.9
19.1
11.1
30.4
27.8
33.9
27.0
TEST
OPTEMIZATION
TENT
77.8
77.2
80.1
73.8
80.6
74.2
74.2
75.5
74.3
73.4
72.0
73.5
77.7
76.7
78.8
76.0
TENT
38.7
35.8
46.5
20.3
41.4
19.1
20.0
21.3
22.8
19.0
11.0
16.6
30.4
27.8
53.9
27.0
COTTA
77.8
75.1
78.4
77.1
78.0
77.2
COTTA!
68.1
63.8
72.7
37.7
73.1
35.1
38.9
43.0
12.4
35.0
16.2
34.6
60.2
52.6
62.6
19.1
LAME
77.8
67.4
63.0
25.4
42.2
12.4
14.2
9.3
26.2
11.2
4.6
34.5
5.9
53.0
6.3
30.6
3.8
4.8
5.3
4.3
NOTE
$7.9
35.3
44.6
18.2
44.4
17.2
19.7
20.1
21.0
18.7
10.5
9.3
31.6
27.7
34.1
26.0
NOTE!
37.5
19.1
42.0
18.9
22.8
19.8
11.3
30.3
26.4
54.4
26.7
RoTTA
37.9
35.6
46.1
18.5
46.2
20.5
10.9
31.5
27.6
35.9
27.3
ROTTA!
37.7
35.1
46.2
19.2
42.1
18.9
18.8
21.5
23.0
19.9
11.3
16.5
30.5
26.7
$4.4
26.8
ROID
75.6
74.9
76.7
70.7
79.2
71.1
71.1
72.7
712
70.0
68.5
70.2
75.7
74,2
76.8
73.4
ROID
24.4
22.5
32.3
8.4
27.9
7.7
7.6
10.3
11.2
9.4
3.9
7.3
17.1
17.7
21.7
15.3
(b) Continual domain non-i.i.d test-time adaptation.
METHOB
GAUSS
SHOT
DMPUL
DEFOC
GLANS
MOTION
200M
SNOW
FROST
FOG
BRRGEL
CONTR.
ELASE
PINIL
278G
ANY.
Sourve
25.1
26.0
26.6
58.5
30.3
43.5
1EST
TIME NORMAL LZATION
THN
77.8
77.2
80.1
73.9
80.6
74.2
74.2
75.5
74.3
73.5
72.01
71.6
77.7
76.7
78.7
76.0
G-BN
60.7
56.3
66.5
41.8
58.0
38.0
40.0
35.4
39.8
31.6
19.2
38.6
41.5
$4.0
44,
44.4
IABN
43.5
40.7
51.2
20.6
48.6
19.5
21.88
22.0
24.0
21.2
11.3
34.1
30.5
36.6
29,1
54.7
UIMIx-TNS
38.9
30.8
41.1
36.0
41.0
21.1
15.3
20.1
18.8
22.2
9.9
18.4
31.2
23.8
33.5
26.8
TEST
TIMS OPTEMIZATION
TENT
77.8
77.2
80.1
73.8
80.6
74.2
74.2
75.4
74.2
73.4
71.9
73.3
77.7
76.5
78.7
75.9
TENT
41.2
21.7
9.9
30.8
23.5
26.6
COTTA
71.8
71.2
80.1
76.3
81.0
77.4
76.5
77.0
96.5
76.0
75.3
79.4
79.2
78.2
79.2
77.8
COTTA!
58.8
71.6
50.8
69.3
32.4
23.8
28.7
26.5
32.9
12.6
50.8
40.9
58.8
44.6
LAME
25.4
4.6
5.9
30.6
LAME
7.8
4.9
6.8
14.9
7.2
6.7
6.1
8.0
8.3
8.2
5.4
9.6
10.1
7.4
8.7
8.0
NOTE
$7.9
30.8
24.1
45.1
18.4
20.1
20.3
19.7
19.3
12.6
33.0
33.2
34.3
26.7
NOTE!
37.5
30.4
41.9
30.7
42.4
21.1
15.5
20.5
19:7
21.4
10.5
19.6
29.5
25.7
33.4
26.7
RoTTA
37.9
33.3
44.1
25.0
46.5
20.7
16.1
22.5
22.2
21.9
10.5
20.6
33.1
27.7
35.7
27.9
19.7
21.6
10.8
30.7
25.3
26.8
ROLD
75.6
74.8
78.5
20.7
79.2
71.1
71.1
72.7
71.2
70.0
68.5
70.2
75.7
74.2
76.8
73.4
ROID
24.4
18.3
28.8
15.9
30.1
88
6.1
10.3
9.9
4.0
10.5
168
18.0
21.0
15.5
(c) Mixed domain non-i.id. test-time adaptation.
PROST
FOG
FLASE
PINEL
Sourve
72.3
65.7
72.9
47.0
$1.3
31.8
42.0
25.1
41.3
26.0
9.3
16.7
26.6
58.5
30.3
43.5
11.51
TRN
80.3
73.8
43.4
84.6
85.0
79.2
O-RN
76.6
$7.8
14.7
42.7
67.6
46.0
$3.0
JABN
43.5
40.7
51.2
20.6
48.6
19.5
21.8
22.0
24.0
21.2
11.3
10.9
34.1
30.5
36.6
29.1
RBN
88.9
$6.5
96.7
78.5
89.2
74.8
75.4
67.5
68.
71.3
41.2
73.3
82.1
84.1
$1.2
77.3
UIMIx-TNS
60.4
55.9
71.5
45.9
49.6
36.2
43.2
25.0
36.8
26.2
12.6
34.4
41.1
59.7
40.1
41.9
TENT
$7.8
80.7
77.5
73.9
$4.5
85.0
79.1
TENT
60.3
55.0
68.7
39.3
46.1
31.3
37.0
21.6
24.6
21.9
10.9
30.8
34.7
58.5
35.01
38.4
CoTTA
91.9
$9.8
96.6
82.1
90.7
80.1
78.3
73.3
73.3
81.4
50.0
90.9
83.7
80.2
78.9
81.4
COTTA!
85.6
83.1
92.0
71.4
82.4
68.1
68.4
62.2
63.8
67.8
40.2
77.9
73.5
76.1
69.8
72.1
LAME
36.1
31.5
29.9
12.4
18.7
9.8
11.6
6.9
16.2
6.8
2.5
14.4
5.8
30.6
7.7
16.1
LAME
5.8
20.4
NOTE
47.8
45.5
54.7
29.0
55.7
26.3
30.9
28.8
28.0
27.8
16.5
14.3
44.4
45.8
45.3
36.1
NOTE!
67.8
63:7
79.9
60.5
60.9
50.6
56.2
33.9
35.8
39.6
18.5
52.3
52.8
66.1
50.2
52.6
ROTTA
73.9
67.3
77.0
26.2
69.3
74.5
64.0
ROTTA!
66.0
61.8
73.8
50.1
53.1
40.9
46.4
27.1
29.5
30.2
13.6
39.9
44.7
61.5
42.1
45.4
ROID
89.3
87.0
97.1
78.2
89.8
74.8
75.1
67.2
68.0
71.7
40.2
72.6
82.7
83.7
81.4
77.3
ROID
53.7
49.9
61.4
29.6
40.5
24.6
28.0
18.9
22.4
19.6
8.6
24.4
28.8
49.1
30.2
32.6
20

---

Published as a conference paper at ICLR 2024
Table 7: Corruption-specifie error rates under three non-i..d. test-time adaptation scenarios. The depicted error rates correspond to the adaptation from CIFAR 100 to CIFARI00 on temporally correlated samples by class labels using Dirichlet distribution with å 0.01 and corruption severity of 5. Methods marked with denote the integration of our proposed UnMix-TNN with the respective TTA method. Averaged over three runs.
(a) Single domain non-i.idd test-time adaptation.
NETHOB
GAUSS
SHOT
INPUL
DRFOC.
GLASS
MOTION
Z0OM
SN0W
PROST
FOG
CONTR.
FLASE
PINEL
JPPG
AVG.
Source
73.0
68.0
39.4
29.3
54.1
30.8
28.8
39.5
45.8
50.3
29.5
55.1
37.2
74.7
41.3
46.5
IBN
79.3
83.3
79.2
50.5
82.3
$3.5
81.6
o-BN
64.0
61.4
50.7
40.5
57.4
42.1
40.3
48.6
19.2
56.1
38.9
19:1
13.8
59.7
52.8
50.7
IABN
65.6
64.2
65.9
47.5
66.2
47.55
47.0
52.5
53.0
64.2
43.4
38.
59.4
57.3
63.9
55.7
RBN
51.3
49.7
51.3
36.9
50.6
38.7
57.2
44.3
43.9
51.2
35.5
39.7
45.2
42.4
50.3
44.6
UIMIx-TNS
48.3
44.0
31.5
46.6
31,0
35.9
46.9
27.6
40.5
40.5
44.9
39.2
TEST
OPTEMIZATION
TENT
83:4
62.8
83.5
79.3
83.9
79.8
79.3
$1.7
81.2
83.3
79.3
80.6
82.2
80.8
83.6
81.6
TENT
47.7
45.5
43.2
30.9
46.2
31.4
30.6
35.7
35.6
46.3
27.4
36.0
40.0
39.6
44.3
38.7
COTTA
79.5
79.2
82.9
79.5
80.9
COTTA!
S7.4
55.8
56.9
11.8
57.3
41.9
41.0
19.5
09.2
58.8
35.4
19.0
53.3
19.0
55.0
50.1
LAME
67.3
56.9
20.0
18.4
34.3
20.0
18.7
29.4
32.8
33.7
19.1
46.1
26.6
76.3
28.7
35.2
31.6
34.7
24.5
23.5
33.0
33.7
53.8
31.7
NOTE
64.1
62.0
62.4
45.2
63.0
45.6
45.1
51.4
50.9
60.8
42.6
31.7
56.7
53.8
60.3
53.4
NOTE!
44.5
30.4
45.4
35.5
45.7
26.8
39.5
37.6
43.9
38.3
RoTTA
50.0
50.5
33.3
48.9
41.4
46.1
31.7
40.6
48.9
43.5
ROTTA!
48.0
45.9
44.6
31.2
46.4
32.3
31.3
35.8
36.
46.4
27.1
40.0
40.5
39.3
45.4
39.4
ROID
60.4
79.7
80.5
74.3
75.0
742
77.8
77.0
80.4
74.2
75.7
78.6
76.6
80.4
77.7
ROID
21.2
18.8
16.2
8.3
18.8
8.6
8.7
11.0
10.9
18.9
6.7
12.9
13.6
16.2
18.5
14.0
(b) Continual domain non-i.i.d test-time adaptation.
METHOB
GAUSS
SHOT
DMPIL
DEFOC
GLANS
MOTION
Z00M
SNOW
FROST
FOG
BRRGEL
CONTR.
ELASE
PINIL
178G
ANY.
Sourve
29.3
25.8
39.5
50.3
55.1
37.2
74.7
16.5
11ST1
TIME NORMALIZATIE EEATIONN
THN
81.4
82.9
83.5
79.3
83.9
79%8
79.3
K1.7
61.2
83.1
79.2
N0.5
N2.3
811.8
83.5
81.6
6-BN
64.0
61.4
50.7
40.5
57.4
42.1
40.3
48.6
49.2
56.1
38.9
49.4
48.8
59.7
52.8
50.7
IABN
65.6
64.2
65.9
47.5
66.2
47.0
52.5
53.0
64.2
43.4
38.1
59.4
57.3
63.9
55.7
38.1
45.5
42.8
50.6
44.9
UIMIx-TNS
48.3
43.8
43.8
34.0
47.5
32.4
29.7
34.8
35.5
46.8
27.4
38.5
43.1
38.7
43.3
39.2
TESt
YIMS OPTEMIZATION
TENT
83,4
62.7
83.4
79.6
83.8
80.1
79.5
87.00
81.5
83.6
79.8
81.4
82.7
81.4
83.9
81.9
TENT
47.7
27.3
41.5
37.3
37.2
COTTA
81.9
81.5
82.2
79.5
82.9
79.9
79.4
$1.5
50.8
83.4
79.4
81.6
82.2
80.0
82.3
81.2
COTTA!
57.4
56.4
57.4
41.5
38.9
49.1
59.7
35.2
53.4
54.1
49.5
55.8
50.4
LAME
56.9
20.0
18.4
33.7
19.1
26.6
76.3
28.7
35.2
LAME
36.4
33.7
33.5
27.6
34.7
26.6
25.7
29.9
30.3
33.3
22.6
27.1
32.6
32.0
52.1
30.6
NOTE
58.5
59.2
48.4
62.9
46.5
45.0
52.5
59.6
44.8
40.4
57.3
57.0
61.6
53.8
NOTE!
46.3
41.4
41.5
31.3
45.2
30.5
28.5
35.1
36.0
45.3
28.5
40.6
41.9
39.8
44.9
38.5
BoTTA
$1.3
51.2
52.3
36.0
50.9
35.6
32.8
41.0
44.7
49.2
30.3
53.1
47.5
42.2
46.5
44.3
31.9
29.7
44.9
26.9
40.2
38.9
ROLD
60.4
79.7
80.5
74.2
81.1
75.0
74.2
77.8
76.9
80.4
74.1
75.7
75.5
76.5
80.4
77.7
ROID
21.2
16.6
15.8
9.6
19.1
83
7.7
10.6
10.7
17.3
6.4
13.5
13,0
15.0
16.2
13.4
(c) Mixed domain non-i..d. test-time adaptation.
BLASE
PINEL
Sourve
73.0
68.0
39.4
29.3
$4.1
30.8
28.8
39.5
45.8
50.3
29.5
55.1
37.2
71,7
41.3
16.5
11.51
TRN
90.7
90.4
97.2
K5.6
95.9
98.3
94.3
O-RN
59.1
47.3
70.9
47.7
73.2
46.2
72.7
60.4
87.6
JABN
65.6
64.2
65.9
47.5
66.2
47.5
47.0
52.5
$3.0
64.2
43.4
38.1
59.4
$7.3
63.9
55.7
RBN
90.9
90.2
83.6
74,9
86.7
75.2
74.4
79.9
79.3
87.7
67.9
$3.0
83.3
93.6
85.0
82.4
UIMIx-TNS
70.1
67.8
44.3
39.8
53.7
39.3
38.7
42.2
43.6
53.8
32.6
49.4
45.3
79.3
51.8
50.1
TENT
90.9
97.5
$6.0
95.9
98.5
96.3
94.3
TENT
75.0
72.0
50.5
34.8
58.9
35.7
34.2
42.0
46.2
53.9
32.2
54.2
42.8
86.5
49.7
51.2
CoTTA
97.2
97.1
97.5
91.2
96.2
91.4
90.1
93.9
93.1
97.4
88.3
97.9
95.2
93.2
94.7
94.3
COTTA!
81.7
80.7
71.8
56.8
71.2
57.2
54.9
62.8
63.0
72.9
50.3
74.7
63.8
71.1
65.5
66.6
LAME
4.7
4.4
3.2
3.2
3.8
3.4
3.4
3.8
3.7
3.9
3.4
3.7
3.6
4.9
3.7
3.8
LAME
4.3
4.5
NOTE
68.0
66.3
65.0
50.1
65.8
50.3
49.6
$1.2
52.5
63.3
46.0
10.4
58.
62.3
63.5
57.0
NOTE!
73.8
71.2
51.8
39.9
$7.8
40.5
39.3
46.6
48.5
58.8
35.2
65.6
46.3
82.1
51.9
53.8
ROTTA
$7.3
71.3
56.5
41.2
65.5
84.7
69.2
65.0
ROTTA!
72.8
71.3
49.3
44.1
56.3
43.1
42.5
44.6
45.8
56.5
33.5
52.8
48.7
81.5
55.4
$3.2
ROID
97.5
97.5
95.4
89.6
96.3
90.0
89.3
92.4
92.6
96.7
84.0
91.4
95.2
97.9
95.5
43.5
ROID
25.5
23.3
7.2
12.1
6.6
7.8
10.1
5.2
10.0
8.1
32.6
11.6
12.3
21

---

Published as a conference paper at ICLR 2024
Table 8: Corruption-specifee error rates under three non-i..d. test-time adaptation scenarios. The depicted error rates correspond to the adaptation from ImageNet to ImageNet-C on temporally correlated samples by class labels using Dirichlet distribution with ő 0.01 and corruption severity of 5. Methods marked with denote the integration of our proposed UnMix-TNS with the respective TTA method. Averaged over three runs.
(a)
Single domain non-i.i.d. test-time adaptation.
NETHOD
GAUSS
SHOT
INPUL
DRFOC.
GLASS
MOTION
Z00M
WNOW
PROST
FOG
CONTR
BLASE
PIXEL
JP5G
AVG.
Source
97.8
97.1
98.2
81.7
89.8
85.2
77.9
83.5
77.1
75.9
41.3
94.5
82.5
79.3
68.5
82.0
13:51
NORMALIZATION
TBN
92.6
16.7
83.2
o-BN
89.3
88.7
88.2
81.6
85.7
81.2
74.1
75,4
71.5
65.6
44.0
38.3
70.1
67.5
64.8
75.7
IABN
91.8
94.8
94.3
91.0
94.3
36.5
$7.3
80.9
81.2
77.0
60.1
85.5
81.22
80.0
82.9
85.0
RBN
86.3
85.7
85.8
86.5
86.2
76.4
65.1
66.6
69.6
56.0
59.3
85.2
59.5
55,2
63.7
71.3
UIMix-TNS
87.2
N5.5
54.7
36.7
89.3
58.2
63.5
70.6
VEST OPTIIIZTTIN
TENT
91.4
90.9
91.2
92.1
91.9
85.8
79.6
80.8
81.3
73.2
62.8
90.3
76.1
73.7
78.2
82.6
TRNT
84.4
72.7
63.1
64.5
67.5
52.9
56.2
89.7
$7.2
51.8
61.7
69.5
COTTA
92.1
76.4
74.1
76.5
82.7
COITA!
89.2
87.5
$8.9
87.3
86.7
73.5
63.5
65.00
67.8
53.6
36.4
$8.3
57.7
52.7
62.5
70.7
LAME
98.6
97.8
98.8
77.9
68.9
83.0
73.0
81.8
72.7
72.8
30.9
93.9
82.1
75.4
61.9
79.3
LAME
85.5
83.5
84.3
82.6
82.0
67.6
55.1
$7.6
61.1
43.6
25.7
87.6
48.6
42.4
54.3
64.1
NOTE
92.6
92.3
91.9
93.5
93.6
$4.4
82.6
77.1
78.0
71.5
56.3
82.8
15.9
73.7
79.5
81.7
NOTE!
85.8
54.9
37.2
58.9
56.1
70.9
RoTTA
86.2
85.5
84.8
87.1
87.1
63.3
67.2
69.2
52.9
35.6
86.7
$7.0
52.1
61.0
70.2
ROTTA
86.9
85.5
86.2
85.9
85.4
74.5
61.9
66.1
68.8
51.5
37,0
90.1
58.6
53.9
63.6
70.8
92.2
75.7
82.4
ROID
87.2
85.5
86.5
83.8
83.9
70.7
59.3
60.3
63.9
45.8
26.5
88.7
52.6
46.4
57.1
66.5
(b) Continual domain non-ii.d test-time adaptation.
METHOB
GAUSS
SHOT
IMPUL.
DEFOC:
GLASS
MOTION
Z00M
SNOW
FROST
FOG
BR3GEL
CONTR.
ELASE
PINTL
J796
AVGL
Scarre
97.8
98.2
81.7
89.8
77.9
77.1
75.9
41.3
82.5
79.3
68.6
82.0
TEST
TIME NORMAL IZATION
THN
91.9
91.5
91.7
92.6
92.3
36.4
SU.2
$1.3
$1.8
74.0
62.8
91.1
10.7
74.4
79.0
83.2
D-BN
89.3
88.7
88.2
81.6
85.7
$1.2
74.1
75.4
71.5
65.6
44.0
88.3
70.
67.5
64.8
75.7
$7.3
85.5
81.2
85.0
RBN
86.3
85.7
85.7
86.7
86.2
76.4
65.
68.6
69.6
56.1
39.3
N6.2
59.5
55.2
63.7
71.3
UIMIs-TNS
67.2
85.4
85.7
85.6
84.7
73.6
64.2
65.7
68.1
54.4
36.6
X8.9
5XL3
53.9
63.7
70.4
VEST
YIME OPTIMIZATIOO
TENT
91.4
90.0
89.6
90.8
90.0
$4.0
75.4
80.2
50.4
73.8
65.4
38.3
75.9
74.3
77.7
82.0
TENT
86.6
84.9
86.4
87.2
87.7
82.3
$1.3
84.7
$8.4
87.6
80.2
98.4
94.2
95.5
97.3
88.2
COTTA
91.1
90.5
90.5
92.0
91.4
85.8
79.6
80.9
$1.3
73.3
62.8
90.5
76.3
73.9
78.2
82.6
COTTA!
87.3
88.5
88.1
66.7
64.88
53.2
36.4
$7.5
71.4
LAME
98.6
97.8
98.8
77.9
88.9
73.0
72.8
30.9
93.9
75.4
61.9
79.3
LAME
85.5
83.2
83.6
82.6
81.5
66.8
54.7
57.3
60.5
43.4
25.7
87.0
48.5
42.4
54.4
63.8
NOTE
76.2
73.8
81.8
NOTE!
87.1
85.6
86.0
85.6
85.4
74.6
64.9
66.2
68.2
55.1
37.2
88.0
58.8
54.5
64.2
70.8
ROTTA
86.2
85.5
84.3
85.8
87.5
75.6
62.2
66.5
67.6
51.8
35.0
80.9
55.5
50.9
58.1
68.9
85.0
53.4
36.4
57.7
53.5
61.5
69.6
ROID
91.5
91.1
91.3
92.2
92.0
85.8
79.4
80.5
81.1
72.6
61.1
90.5
75.7
73.2
78.0
82.4
ROID
87.2
83.8
45.7
26.6
$2.6
45.9
57.4
66.4
(c) Mixed domain non-i.i.d test-time adaptation.
Source
97.8
97.1
98.2
81.7
89.8
85.2
77.9
83.5
77.1
75.9
41.3
94.5
82.5
79.3
68.6
82.0
1EST
TIME NORMAL EXATTIONO
TBN
99.4
99.1
99.3
98.6
99.0
95.
97.1
96.9
96.5
96.3
81.9
99.2
97.3
95.4
94.2
96.6
96.2
89.3
52.5
81.7
86.7
TABN
94.8
94.0
94.3
86.5
87.3
WIL
81.2
77.0
60.1
N5.5
81.2
80.0
82.9
85.0
RBN
97.8
96.9
97.6
05.3
96.0
94.9
91.5
91.2
90.3
89.6
64.3
97.4
92.5
87.1
$4.00
91.1
UnMIs-TNS
96.7
95.6
96.4
90.6
90.5
87.1
$1.3
82.5
82.2
78.0
48.3
94.2
83.4
83.0
75.0
84.3
TENT
99.2
97.9
97.9
82.2
98.5
98.2
97.5
97.6
TENT!
99.8
99.7
99.8 8
97.6
96.6
95.5
92.8
94.3
94.0
91.6
77.3
99.2
94.5
97.0
95.8
45.0
COITA
99.6
99.4
99.5
98.8
98.9
98..3
97.1
96.9
96.6
96.4
81.4
999.
97.3
95.9
94.77
96.7
COTTA!
99.6
99.3
99.5
91.2
90.7
87.2
$1.4
82.2
81.9
78.5
47.7
95.3
83.3
86.7
78.6
85.6
LAME
85.9
85.4
65.9
$8.6
70.0
65.5
58.9
65.7
59.2
57.8
29.0
74.5
66.6
608.8
50.3
65.0
LAME
83.4
84.4
73.2
72.8
69.3
63.3
64.2
65.1
60.3
35.3
80.5
66.2
67.3
57.5
68.5
NOTE
95.7
65.4
82.0
83.8
85.6
NOTE!
96.9
95.9
96.7
90.9
90.7
87.4
81.6
82.8
82.5
78.4
49.0
94.3
83.8
83.2
75.7
84.6
ROTTA
94,8
94.3
94.7
92.5
93.9
83.2
$6.1
87.0
50.6
96.6
88.3
79.5
72.3
86.7
ROTTA!
96.8
96.
96.5
90.3
00.5
87.2
81.7
82.4
81.9
78.
44.7
94.8
83.0
81.3
73.7
83.9
ROID
99.4
99.1
99.3
98.6
98.9
98.3
97.0
96.8
96.5
96.2
81.4
99.2
97.2
95.5
94.2
96.5
ROID
40.6
76.0
77.9
69.3
17.5
22


Reviews:
[{"review_id": "0WQtg24haE", "reviewer": "Reviewer_Yi31", "summary": "This paper studies how to adapt the batch norm layers for test-time adaptation. Standard TBN method assumes that test samples are i.i.d. sampled from a single target distribution, while this assumption can be violated in real scenarios: samples can be drawn from multiple distributions and can be temporal correlated. The author propose a nuanced method called UnMix-TNS, which split the running statistics of each BN layer into K different components. For each testing sample, only the closest component will be refined, which makes the BN layer more stable and robust to changes in batch-wise label distribution. The author verify the proposed method over benchmarking datasets, several settings, a wide range of models, and compare to a variety of baselines.", "questions": "Besides the major weaknesses, there are several minor questions: \n1. In Equation (5), it seems like all $\\mu_{k, c}^0$ for different $k$ distribute on the line of $\\mu_c + t \\sigma_c$. Is that intentional? Or it makes more sense if they do not have such low-rank structure? \n2. Equation (7). Are there any insight on using cosine similarity instead of L2 distance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Clear motivation: The author clearly summarizes key drawbacks of BN-based TTA methods. Figure 1 is intuitive and informative. \n2. Algorithm design: The algorithm is generally intuitive to me. Clearly if each component corresponds to a label, the algorithm is likely to be robust to the bias in label distribution. Also the author pay special attention on the initialization of UnMix-TNS components to preserve the statistical properties. \n3. Experiments: The proposed method is tested on a variety of datasets, models, evaluation protocols, and is compared to a wide range of related baselines.", "weaknesses": "Major (These weaknesses or concerns significantly affect my understanding and decision regarding this paper)\n\n1. Multiple target domains: One of the claimed contribution is that UnMix-TNS has robustness when tested on continual domain and mixed domain. However, I am very confused which part of UniMix-TNS is designed for and beneficial to these multi-domain settings, especially the mixed domain setting. Considering that there is a new batch containing images from different domains, although different images may correspond to different UnMix-TNS components, they are finally normalized with the same mean and variance according to Eq (11). It seems like UnMix-TNS still treat multiple target domains as one single target domain. \n2. Unclear experimental setting and unsatisfactory performance: The author claimed that they follow the protocol outlined in (Lim et al., 2023). However, the performance for most of the baselines are significantly worse than the results in (Lim et al., 2023). Also, the proposed UnMix-TNS fails to outperform “Source” in mixed domains. I believe the comparison to baselines only makes sense if the test-time normalization is beneficial. (I presume that “Source” means no adaptation. Please correct me if I am wrong)\n3. If the temporal correlation mainly \n4. Choice of K. The appendix discusses the influence of K. However, it is still unclear how to choose K in practice. How is it related to the number of classes and number of domains? (Although the number of domains might not be exposed.)\n\nMinor (These minor weaknesses are not crucial but I believe fixing them will improve the quality of the paper) \n1. Figure 2 is pixelated. Please consider improving the dpi. \n2. The temporal correlation might be better explained in Section 2.1. Does it refer to correlation of feature, label, or domain? \n3. Page 5 after Equation (1), two exp have different font style. Also I recommend changing $\\sim$ to $\\approx$ since $\\sim$ usually means “following the distribution of”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fr2MW9P5jr", "reviewer": "Reviewer_auPw", "summary": "The paper proposes Un-Mixing Test-Time Normalization Statistics (UnMix-TNS), a novel BatchNorm adaptation strategy under the assumption that the test distribution is a shifting mixture. Empirical experiments on multiple CV tasks indicate that UnMix-TNS can adapt to the temporal correlation of unlabeled test streams.", "questions": "In Appendix B.2, it's unclear what `torch.var_mean(x, dim=[2, 3])` means. Does `x` have dimension `(batch, channel, height, weight)`?\n\nFrom my understanding, UnMix-TNS will fail when test samples gradually shift from one cluster to another, in which case the active $\\mu_k$ would follow the sample, causing two clusters to eventually overlap. How would you prevent that?\n\nNot really a question, but an interesting experiment is to set $K$ to the number of corruption types, and see if $p_{b,k}^t$ can recover the corruption type for each sample correctly. Maybe each cluster will correspond to the label instead of corruption. Who knows?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Novel idea!\n\nIntuitive toy example (i.e. Figure 1) illustrating the motivation of the proposed method.\n\nGood empirical performance, improving over baseline methods most of the time. Mixes well (no pun intended) with other BN adaptation methods like Tent and LAME, making them even better. Synergy!\n\nNegligible additional inference time cost (the authors should highlight it instead of burying it under a subsection titled \"Ablation Studies\").", "weaknesses": "UnMix-TNS seems tightly coupled to the convolutional layers, as it considers the channel dimension $C$ and all experiments are computer vision tasks. It is unclear whether it will work for other deep learning tasks, e.g. NLP.\n\nThe paper doesn't discuss how to choose $K$ and $\\lambda$, and the additional experiments in Appendix C only study the sensitivity to $K$.\n\nAlgorithm 1 in Appendix B.2 is both helpful and confusing: helpful because it describes the algorithm concisely; but confusing because the shapes don't add up. For example, `instance_mean` should have shape `(B, 1, C)`, otherwise you cannot get `hat_mean` of size `(B, K, C)` through broadcasting. It would be better to include a runnable Python snippet in the Appendix, even if that's slightly more verbose.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "auGxJrhucy", "reviewer": "Reviewer_yuas", "summary": "This paper introduces UnMix-TNS, a new approach to address the challenges of test-time adaptation in neural networks under non-i.i.d. conditions, particularly focusing on data streams with temporal correlation. Traditional batch normalization (BN) techniques, fundamental in stabilizing neural network training, fall short under non-i.i.d. test scenarios, often encountered in real-world applications like autonomous driving. UnMix-TNS innovatively recalibrates instance-wise statistics within BN by mixing them with multiple unmixed components, simulating an i.i.d. environment. This method is adaptable to various state-of-the-art test-time adaptation strategies and seamlessly integrates with pre-trained architectures that include BN layers. The approach is empirically validated across diverse scenarios and shows promise in effectively handling data with temporal correlations. New datasets have also been introduced.", "questions": "- Could the authors provide more detailed theoretical insights or foundational principles on how UnMix-TNS adapts to test data streams with temporal correlation? Specifically, how does the method theoretically account for and mitigate issues that arise due to temporal dependencies in the data?\n- What is the theoretical rationale behind the 'unmixing' approach in handling temporally correlated data? How does this strategy compare with traditional methods in terms of theoretical robustness against temporal variations in data streams?\n- Could the authors elaborate on the criteria or algorithm used to select instances from incoming test batches for updating the UnMix-TNS statistics? How do the authors ensure that this selection process does not introduce bias, which might affect the model’s generalization capability?\n- In situations where the test batches contain highly diverse or outlier instances, how does UnMix-TNS maintain the integrity of the normalization statistics? Is there a mechanism in place to detect and appropriately handle such anomalies?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. **Innovative Approach to Test-Time Adaptation:** The introduction of UnMix-TNS recalibrates instance-wise statistics by mixing with multiple unmixed statistics components. It targets the challenge of non-i.i.d. test batch distributions, a limitation in current test-time adaptation methods.\n2. **Seamless Integration with Pre-Trained Models:** The ability of UnMix-TNS to integrate with existing pre-trained neural network architectures without significant modifications is also an advantage. This facilitates easier adoption in various applications.\n3. **Empirical Validation:** The paper provides empirical evidence of performance improvements across multiple benchmarks, including those with common corruption shifts and natural shifts. UnMix-TNS demonstrates robustness in diverse scenarios. The introduction of new datasets like ImageNet-VID-C and LaSOT-C for realistic domain shifts in frame-wise video classification adds value.", "weaknesses": "1. **Lack of Theoretical Insight for Temporal Correlation Handling:** The abstract and introduction do not provide clear theoretical insights or explanations on how UnMix-TNS effectively deals with test data streams having temporal correlation. A deeper understanding of the underlying principles is crucial for assessing the robustness and reliability of the method.\n2. **Potential for Bias in Instance Selection:** The effectiveness of UnMix-TNS might be influenced by the selection of instances from incoming test batches for updating statistics. If the selection is not well-designed, it could introduce bias, affecting the generalization of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mOA5uHpAX2", "reviewer": "Reviewer_omE5", "summary": "This paper proposes a novel test-time normalization layer to tackle the temporally correlated, distributionally shifted problems within the context of test-time adaptation, and boost the performance of test-time adaptation tasks involving various distribution shifts. The experiments demonstrate the method achieved SOTA.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The logic of this paper is clear and the performance is excellent.\nThe author argues that the assumption of test batches conforming to an i.i.d distribution can produce unstable and unreliable adaptations. This paper works on it and offers a reasonable approach.  \n2. Extensive experiments on three benchmarks and solid ablation studies.\n3. The paper is well organized and the idea is well presented.", "weaknesses": "After reading the paper, it is clear that the method delivers improvements on TTA tasks. However, there are some questions remain unanswered:\n0. Some papers also explored the test-time non-iid setting, such as [a] and [b]. Compared with these papers, what's the contribution of the proposed setting and approach?\n1. Why the hyper-parameter \\alpha is set to be 0.5 in Equation (5) for implementation while initializing UNMIX-TNS components?\n2. In Section 2.2.3, the hyper-parameter \\lambda is not explained clearly.\n3. While Tables 1 and 2 include 11 methods for comparison, the experiments on the video dataset (Table 3) only include seven methods.\n4. Does the UNMIX-TNS update the same parameters with other TTA methods? And have the affine parameters in BN layers been changed within the optimization? Does the method require only single forward inference or multiple forward passes?\n5. The authors mentioned that “only components that closely align with the instance-wise statistics undergo updates”, but equations (12) and (13) update all K BN statistics components with different assignment probabilities. \n\n\n[a] Robust Test-Time Adaptation in Dynamic Scenarios, ICCV2023\n[b] Robust continual test-time adaptation: Instance-aware BN and prediction-balanced memory. NeurIPS2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)