TTN: A Domain-Shift Aware Batch Normalization in Test-Time Adaptation
Authors: Hyesu Lim
Byeonggeun Kim
Jaegul Choo
Sungha Choi
Venue: ICLR 2023 poster (2023)
Keywords: Test time adaptation
Domain adaptation
Batch Normalization

Abstract: This paper proposes a novel batch normalization strategy for test-time adaptation. Recent test-time adaptation methods heavily rely on the modified batch normalization, i.e., transductive batch normalization (TBN), which calculates the mean and the variance from the current test batch rather than using the running mean and variance obtained from the source data, i.e., conventional batch normalization (CBN). Adopting TBN that employs test batch statistics mitigates the performance degradation caused by the domain shift. However, re-estimating normalization statistics using test data depends on impractical assumptions that a test batch should be large enough and be drawn from i.i.d. stream, and we observed that the previous methods with TBN show critical performance drop without the assumptions. In this paper, we identify that CBN and TBN are in a trade-off relationship and present a new test-time normalization (TTN) method that interpolates the statistics by adjusting the importance between CBN and TBN according to the domain-shift sensitivity of each BN layer. Our proposed TTN improves model robustness to shifted domains across a wide range of batch sizes and in various realistic evaluation scenarios. TTN is widely applicable to other test-time adaptation methods that rely on updating model parameters via backpropagation. We demonstrate that adopting TTN further improves their performance and achieves state-of-the-art performance in various standard benchmarks.

Published as a conference paper at ICLR 2023
TTN: A DOMAIN-SHIP AWARE BATCH NORMALIZA- TION IN TEST TIME ADAPTATION
Hyesu Lim 12, Byeonggeun Kim", Jaegul Choo,, Sungha Choil Qualcoom Al Research", 2KAIST
ABSTRACT
This paper proposes a novel batch normalization strategy for test-time adaptation. Recent test-time adaptation methods heavily rely the modified batch normal- ization, i.e. transductive batch normalization (TBN), which calculates the mean and the variance from the current test batch rather than using the running mean and variance obtained from source data, i.e., conventional batch normalization (CBN). Adopting TBN that employs test batch statistics mitigates the performance degra- dation caused by the domain shift. However, re-estimating normalization statistics using test data depends on impractical assumptions that test batch should be large enough and be drawn from i.i.d. stream, and we observed that the previous meth- ods with TBN show critical performance drop without the assumptions. In this paper, we identify that CBN and TBN are in a trade-off relationship and present a new test-time normalization (TTN) method that interpolates the standardization statistics by adjusting the importance between CBN and TBN according to the domain-shift sensitivity of each BN layer. Our proposed TTN improves model robustness to shifted domains across wide range of batch sizes and in various realistic evaluation scenarios. TTN is widely applicable to other test-time adap- tation methods that rely updating model parameters via backpropagation. We demonstrate that adopting TTN further improves their performance and achieves state-of-the-an performance in various standard benchmarks.
INTRODUCTION
When we deploy deep neural networks (DNNs) trained on the source domain into test environments (i.e., target domains), the model performance on the target domain deteriorates due to the domain shift from the source domain. For instance, in autonomous driving, a well-trained DNNs model may exhibit significant performance degradation at test time due to environmental changes, such as camera sensors, weather, and region (Choi et al., 202 Lee ct al. 2022 Kim et al.) 2022b).
Test-time adaptation (TTA) has emerged to tackle the distribution shift between source and target domains during test time (Sun et al. 2020;; Wang et al.) 2020). Recent TTA approaches (Wang ett al., 2020; Choi et al. 2022: Liu a (2021) address this issue by 1) (re- (re-)estimatin normaliza- tion statistics from current test input and 2) optimizing model parameters in unsupervised manner, such entropy minimization (Grandvaler & Bengio, 2004 Long al. 2016 Vu al., (019) and self-supervised losses (Sun et al., 2020 Liu et al. (021)). In particular, the former focused on the weakness of conventional batch normalization (CBN) (Loffe Szegedy 2015) for domain shift in test time. As described in Fig. [b), when standardizing target feature activations using source statistics, which are collected from the training data, the activations can be transformed into an un- intended feature space, resulting in misclassification. To this end, the TTA approaches (Wang et al.) 2020 Choi et al. 2022; Wang et al. (0022) have heavily depended on the direct use test batch statistics to fix such an invalid transformation in BN layers, called transductive BN (TBN) (Nado
ct al. 2020 Schneider et al. 2020 Bronskill et al. 2020 (sce Fig. [()..
The approaches utilizing TBN showed promising results but have mainly been assessed in limited evaluation settings (Wang et al. 2020 Choi et al. 2022. Liu et al. (02T). For instance, such evalua- tion settings assume large test batch sizes (e.g., 200 more) and a single stationary distribution shift
Work completed while at Qualcomm Technologies, Inc. 'Corresponding author. "Qullcomm AI Research initiative of Qualcomm Technologies, Inc.

---

Published as a conference paper at ICLR 2023
Standardize
Source features
Test features
Class
Class
Source mean
x Test meon
(a) Valid output using CBN
(b) Invalid output using CBN
(c) V Valid output using TEN
100
Test batch size
200
64
Enorrata (%)
80
60
40
0
CBN
TBN
TINOurS
TENT TENT (T SWR
SWR+ STNOOS)
(c) Performance drops in small test batches using TEN
Figure 1: Trade-off between CBN & TBN. In conceptual illustrations (a), (b), and (c), the depicted standardization only considers making the feature distribution have zero mean, disregarding mak- ing it have unit variance. When the source and test distributions are different, and the test batch size is large, (b) test features can be wrongly standardized when using CBN (Toffe & Szegedy 2015). but (c) TBN (Nado al., 2020) can provide valid output. (d) Error rates (3) shifted domains (CIFAR- 10-C). TBN and TBN applied (TENT (Wang et al. (020). SWR (Choi et al. (2022) meth- ods suffer from severe performance drop when the batch size becomes small, while TTN (Ours) improves overall performance.
(i.e., single corruption). Recent studies suggest more practical evaluation scenarios based on small batch sizes (Mirza et al. 2022 Hu al., 2021 Khurana al. 202T) or continuously changing data
distribution during test time (Wang et al. 2022). We show that the performance of existing methods
significantly drops once their impractical assumptions of the evaluation settings are violated. For example, shown in Fig. [d). TBN (Nado et al., (020) and TBN applied methods suffer from severe performance drop when the test batch size becomes small, while CBN is irrelevant to the test batch sizes. We identify that CBN and TBN are in trade-off relationship (Fig. D in the sense that one of each shows its strength when the other falls apart.
To tackle this problem, present a novel test-time normalization (TTN) strategy that controls the trade-off between CBN and TBN by adjusting the importance of source and test batch statistics according the doiain-shifi sensitivity of each BN layer. Intuitively, we linearly interpolate be- tween CBN and TBN so that TBN has a larger weight than CBN the standardization needs to be adapted toward the test data. We optimize the interpolating weight after the pre-training but before the test time, which we refer to as the post-training phase. Specifically, given a pre-trained model, first estimate channel-wise sensitivity of the affine parameters in BN layers domain shift by analyzing the gradients from the packppropggttion of two input images, clean input and its aug- mented one (simulating unseen distribution). Afterward, we optimize the interpolating weight using the channel-wise sensitivity replacing BN with the TTN layers. It is noteworthy that none of the pre-trained model weights are modified, but we only train newly added interpolating weight.
We empirically show that TTN outperforms existing TTA methods in realistic evaluation settings, i.., with a wide range of test batch sizes for single, mixed, and continuously changing domain adaptation through extensive experiments image classification and semantic segmentation tasks. TTN as a stand-alone method shows compatible results with the state-of-the-ar methods and com- bining our TTN with the baselines even boosts their performance in overall scenarios. Moreover, TTN applied methods flexibly adapt to new target domains while sufficiently preserving the source knowledge. No action other than computing per batch statistics (which can be done simultaneously to the inference) is needed in test-time; TTN is compatible with other TTA methods without requir- ing additional computation cost.
Our contributions are summarized as follows:
We propose a novel domain-shinf aware test-time normalization (TTN) layer that combines source and test batch statistics using channel Saannel-wise interpolating weights considering the sensitivity to domain shift in order to flexibly adapt to new target domains while preserving the well-trained source knowledge.
2

---

Published as a conference paper at ICLR 2023
Pre-train (CBN)
(a-1) Post-train (TTN)
Test time (TTN)
Per-batch
Prozen
Optimize
Standaddize
Alline Transform
Zin
Zout y 2 +
(a) Overall procedure of train and test phases
CBN in test
TBN best time
(b-1) (TN() in post-train
(TN()UUS test time
Zin
T
Zout
Zin
Bout
Zin
Zout
-
Zout
YBB
a
Y.B
33.
Y.B
Y,B
Pire
M5 OS
Pin Oin
Us. de
(b) Comparison of BN layers
Figure 2: Method overview. (a) We introduce an additional training phase between pre-train and test time called (a-1) post-training phase. (b) Our proposed TTN layer combines per-batch statistics and frozen source statistics with interpolating weight a, which is (b-1) optimized in post-training phase and (b-2) fixed in test time.
To show the broad applicability of our proposed TTN, which does alter training or test- time schemes, we show that adding TTN to existing TTA methods significantly improves the performance across a wide range of test batch sizes (from 200 and in three realistic evaluation scenarios; stationary, continuously changing, and mixed domain adaptation.
We evaluate our method through extensive experiments CIFAR-IO/100-1 and ImageNet-C (Hendrycks & Dietterich 2018) and semantic segmen- tation task using CityScapes (Cordts | BDD 100K (Yu et al. 2020), Mapil- lary (Neuhold et al. (017). GTAV (Richter et al. 2016), and SYNTHIA (Ros et al. 2016).
image classification using
2 METHODOLOGY
In this section, we describe our method, the Test-Time Normalization (TTN) layer, whose design is suitable for test-time adaptation (TTA) in practical usages out of the large batch size and i.i.d assumptions during a test time. We first define the problem setup Section 2.1 posed TTN layers in Section
2.1 and present our pro-
2.2 Finally, we discuss how we optimize TTN layers in Sections 2.3
2.1 PROBLEM SETUP
Let the train and test data be Dss and Df and the corresponding probability distributions be Ps and Pr, respectively, where Ds and DD share the output space, i.e., {3r ~ Ds = {3/ ~ Dy. The covariate shift in TTA is defined as Ps(x)) Pr(x:) where Ps(y)æ) Pr (r) Cainonerr-CCander
al. 2008). A model, fe, with parameters 0, trained with mini-batch. E { ) [89
from source data Ds, where Tú is an example and yi is the corresponding label. During the test, So encounters test batch B" Dr, and the objective of TTA is correctly managing the test batch from the different distribution.
To simulate more practical TTA, we mainly consider two modifications: (1) various batch sizes, [B where small batch size indicates small latency while handling the test data online, and (2) multi, Nttarget domains. DT {Dr, - Under this setting, each test batch D" is drawn by one of the test domains in Dr, where Dr may consist of a single target domain, multiple target domains, or mixture of target domains.
2.2 TEST-TIME NORMALIZATION LAYER
We denote input of BN layer as E IR BCHW forming mini-batch size of B. The mean and variance are j and o², respectively, which are computed as follows:
B H
W
W ΣΣ>
B
Me BHW ΣΣΣ
Zbckws
0²
(Zbchw - -Ma²
(1)
BHW
10
where μ and 0² are in RC, and C. H. and W stand for the number of channels. dimension of height, and that of width, respectively. Based on and o², the source statistics Ps. 03 E RC are usually estimated with exponential moving average over the training data.
3

---

Obtain Prior A. To identify which BN layers and corresponding channels are sensitive domain shifts, we simulate the domain shifts by augmenting"ite clean image, i.e., original training data, and make a pair of (clean æ, analyze in which layer and channel the standardization statistics should be corrected, we consider the standardized features z(f.c) of 2(1,)) for a channel index at a layer I, whose input is clean r. We compare g(L.c) to that of domain-shifted one, z/(Le) from x'. Since the pre-trained CBN uses Published as a conference paper at ICLR 2023
Per Per-hatcc statistics
Prozen statistics
Gradient flow
1
H5
1/
CAR
CONN
CEN
ReL
C
Rell
LEE
TIN
(a-2) Prior A
1
(b-2) Optimize a
(-11) Gradient of affine parameters y.B
v(l
(b-1) Initialize with prior A
CorN
CEN
C
LcB
Conv
Len + LMSE
TIN
TIN (b) Optimize a
(a) Obtain prior
Figure 3: Two stages in post-training phase. (a) Given pre-trained model, which uses CBN, and its training data, we obtain prior knowledge of each BN layer. (a-1) We first compute gradients of affine parameters in each BN layer from clean and augmented input x' and obtain the gradient distance score (Eq. 4.. (a-2) For BN layers with larger distance score, we put more importance on current batch statistics than source statistics (i.e., large a), and define prior A accordingly (Fq.5). (b) After obtaining prior A, we substitute BN layers from CBN to TTN. (b-1) Initializing a with prior A, (b-2) optimize using CE and MSE loss (Bq. 6 with augmented training data x'.
In BN layers, input is first standardized with statistics μ and o² and then is scaled and shifted with learnable parameters y and in RCC The standardization uses current input batch statistics during training and uses estimated source statistics Hı, and o² at test time (Fig. 2)).. To address domain shifts in test time, we adjust the source statistics by combining the source and the test mini-batch statistics (Singh & Shrivastava 2019 Summmers & Dinneen 2019) with learnable interpolating weight a E R ranges [0, 1I. Precisely, TTN standardizes feature with
a= + (1 - a))ss === + (1 a)) - a(1 aa ())-))) ms)²
(2)
while using the same affine parameters, y and B. Note that we have different mixing ratios a, for every layer and channel.
2.3 POST TRAINING
Like Choi al. (022). we introduce additional training phase, the post-training (after pre- training but before testing), to optimize the mixing parameters a in Eq. 2 all parameters except a frozen and we have access the labeled source data during the post- training. We first obtain prior knowledge A of a by identifying which layers and their channels are sensitive domain shifts. Then, optimize a with the prior knowledge and an additional objective term. The overall procedure is depicted in Fig. 3 and the pseudocode is provided in appendix A.3
(Fig. aa). Note that
x') images, where the semantic information is shared. To
the same (1,c) and (l.c) for both inputs, the difference between 2(t.c) and g/(l.c) is caused by the domain discrepancy between x and x'. We argue that if the difference significant, the parameter at (1, c) is sensitive to the domain shift, i.e., intensely affected by the domain shift, and hence the standardization statistics at (1, c) should be adapted towards the shifted input.
Drawing inspiration from Choi al. (2022). measure the domain-shift sensitivity by comparing scaled and shifted by and / in each BN layer, we compare the gradients of affine parameters 7 and B, V., and Va, respectively, measure the dissimilarity of and E'. As described in Fig. 3 l),, we collect the V., and V using cross-entropy
gradients. Since the standardized feature
It is noteworthy that the post-training phase is robust to the choice of data augmentation types. Ablation study results and discussions are provided in the appendix B.4
4

---

Published as a conference paper at ICLR 2023
loss. LCE. To this end, we introduce a gradient distance score, d( I as follows:
R for each channel at layer
9t g
(3)
N
i=1 g |191111
d(e)
(57
(1,c) + 88
(1.r))
(4)
2
where (g.g") (V!..,, v4.00) and (V (1,c)
(gre) for s
(1.c) and $ (l.c)
respectively, N is the number
of training data, and the resulting d(l.e) E [o, 1]. Once we obtain 8y and 83 from Eq. 3 we conduct
min-max normalization over all By
(L.c)
and $ (1.c) before computing Eq.
To magnify the relative difference, take the square final step and denote the result prior A (Fig. 2-22)):
A= (d d(...) dlc.))3
where d(..) [dtl.)). Je=1
Optimize a. The goal of optimizing is to make the combined statistics correctly standardize the features when the input is sampled from an arbitrary target domain. After obtaining the prior A, we replace CBN with TTN layers while keeping the affine parameters. Then, we initialize the interpolating weights a with A, which represents in which layer and channel the standardization statistics need to be adapted using test batch statistics (see Fig. 1()-))). To simulate distribution shifts, we use augmented training data. Expecting the model to make consistent predictions either given clean or augmented inputs, we use cross-entropy loss LCE. Furthermore, to prevent a from moving too far from the initial value A. prior A, i.e,, LMSE = ||a-AAl All as a constraint. Total loss L can be written as L Lee AA (6), where À is weighting hyperparameter (Details provided in the appendix A. B.1
mean-squared error loss LMsE between and the
EXPERIMENTS
In image classification, evaluate TTN for corruption robustness in realistic evaluation settings, ie., where the test batch size can be variant and where the target domain can be either stationary, continuously changing, mixed with multiple domains. Additionally, we further validate TTN on domain generalization benchmarks incorporating natural domain shifts (e.g., changes in camera sensors, weather, time, and region) in semantic segmentation.
3.1 EXPERIMENTAL SETUP
Given models pre-trained on clean source data, we optimize TTN parameter a with the augmented source data the post-training phase. Afterward, we evaluate our post-trained model on the cor- rupted target data. Implementation details provided in the appendix
Datasets and models. We use corruption benehmark datasets CIFAR- 10/1R--0/00--CC and ImageNet-C. which consist of 15 types of common corruptions at five severity levels (Hendrycks & Dietterich) 2018). Each corruption is applied to test images of the clean datasets CCIFAR-10/100 and Ima- geNet). We use training set of the clean dataset for post-training and the corrupted dataset for evaluation. As backbone models, we used WideRestNet-40-2 (Hendrycks et al. 2019) trained on CIFAR-10/100, and ResNet-50 (He et al. 2016) trained on ImageNet. To validate our method in mantic segmentation, we conduct experiments on Cityscapes (Cordts et al. 2016), BDD-100K (Yu al. 2020). Mapillary (Neuhold et al., (017). GTAV (Richter et al. 2016), and SYNTHIA (Ros et al. 2016) datasets, in accordance with the experimental setup for domain generalization proposed in RobustNet (Choi al. 202T).
Baselines. To demonstrate that TTN successfully controls the trade-off between CBN and TBN, we
compare TTN with (1) AdaptiveBN (Schneider et al., 2020). (2) a-BN (You et al. 2021) and (3) MixNorm (Hu et al. 202I). which combines takes the moving average of the source and the test batch statistics with a pre-defined byperparameter (i.e., a constant a). The following baselines are suggested on top of TBN (Nado et al.) 2020); (4) TENT (Wang al. 2020) optimizes BN affine parameters via entropy minimization. (5) SWR (Choi et al. 2002) updates the entire model parame-
ters considering the domain-shif sensitivity. (6) CoTTA (Wang at. 2022 ensembles the output of
5

---

Published as a conference paper at ICLR 2023
Table 1: Single domain adaptation on corruption benchmark. Error rate (4) averaged over 15 corruptions with severity level 5 using WideResNet-40-2 as backbone for each test batch size. We used reported results of MixNorm with fixed parameters from the original paper and denoted as s. we provide variants of TTN, which show stronger performance for small test batch.
In appendix B.3
CIFAR-I0-C
CIFAR-100-C
Method
200
64
Avg.
16
Avg.
Source (CBN)
18.27
18.27
18.27
18.27
18.27
18.27
18.27
46.75
46.75
46.75
46.75
46.75
46.75
46.75
TBN
14.49
15.02
17.10
26,28
35.65
90.00
33.09
39.25
40.21
44.03
59.10
80.65
99.04
60.38
Norm
AdaptiveBN
12.21
12.31
12.89
14.51
15.79
16.14
13.98
36.56
36.85
38.19
41.18
43.26
44.01
40.01
a-BN
13.78
13.77
13.89
14.54
15.16
15.47
14.44
39.72
39.85
39.99
41.34
42.66
45.64
41.53
MixNorm"
13.85
14.41
14.23
14.60 (B=5)
15.09
14.44
Ours (TTN)
11.67
11.80
12.13
13,93
15.83
17.99
13.89
35,58
35.84
36.73
41.08
46.67
57.71
42.27
TENT
12.08
14.78
16.90
25.61
35.69
90.00
32,51
35,52
39,90
43.78
59.02
80.68
99.02
59.65
Optin.
+Ours (TIN)
11.28
11.52
12.04
13.95
15.84
17.94
13.77
35.16
35.57
36.55
41.18
46.63
58.33
42.24
SWR
10,26
13.51
16.61
27.33
40.48
90.04
33.04
32.68
37.41
43.15
59.90
87.07
99.05
59.88
+Ours (TTN)
9.92
11.77
13.41
18.02
24.09
61.56
23.13
32.86
35.13
38.66
49.80
60.72
80.90
49.68
Table 2: Continuously changing domain adaptation on corruption benchmark. Error rate ())
averaged over 15 corruptions with severity level 5 using WideResNet-40-244 as backbone for each test batch size. We omitted 'Nomm' methods results in this table since they are eqaul to that of Table T
CIFAR-IO-C
CIFAR-100-C
Method
200
64
16
Avg.
200
64
16
2
Avg.
Source (CBN)
||
118.27
18.27
18.27
18.27
18.27
18.27
18:27
46.75
46.75
46,75
46.75
46.75
46.75
46.75
Ours (TTN)
111,67
11.80
12.13
13.93
15.83
17.99
13.89
35,58
35.84
36,73
41.08
46.67
57.71
42.27
CoTTA
|| 12.446
14.60
21.26
45.69
58.87
90.00
40.48
39.75
42.20
52.94
73.69
87.66
98.99
65.87
Optin.
TENT
12.54
13.52
15.69
26.23
35.77
90.00
32.29
36,11
37.90
43.78
58.71
81.76
99.04
59.55
+Ours (TTN)
11.44
11.60
12.08
16.14
18.36
22.40
15.33
43.50
37.60
38.28
44.60
54.29
80.63
49.82
SWR
11.04
11.53
13.90
23.99
34.02
90.00
30.75
34.16
35.79
40.71
58.15
15
80.55
99.03
62.56
+Ours (TTN)
10.09
10.51
11.28
14.29
16.67
84.12
24.49
33.09
34.07
36.15
42.41
53.63
93.08
48.74
augmented test inputs, updates the entire model parameters using a consistency loss between student and teacher models, and stochastically restores the pre-trained model. We refer to TBN, (1), (2), and (3) as normalization baasd methods (Norm), the other as optimization- based methods (Optim.), and denote the pre-trainct model using CBN as 'source'.
Evaluation scenarios. To show that TTN performs robust on various test batch sizes, we conduct experiments with test batch sizes of 200, 64, 16, 4, 2, and 1. We evaluate our method in three evalu- ation scenarios; single, continuously changing, and mixed domain adaptation. In the single domain adaptation, the model is optimized for one corruption type and then reset before adapting to the subsequent corruption, following the evaluation setting from TENT and SWR. In the continuously changing adaptation (Wang et al. 2022), the model is continuously adapted to 15 corruption types (w/o the reset), which is more realistic because it is impractical to precisely indicate when the data distribution has shifted in the real world. Finally, to simulate the non-stationary target domain where various domains coexist, we evaluate methods in the mixed domain adaptation setting, where a sin- gle batch contains multiple domains. We use a severity level of 5 (Hendrycks & Dietterich] 2018) for all experiments. It is noteworthy that we use a single checkpoint of TTN parameter for each dataset across all experimental settings.
3.2 EXPERIMENTS ON IMAGE CLASSIFICATION
Tables D 22 and 3 show error rates on corruption benchmark datasets in three different evaluation scenarios; single domain, continuously changing, and mixed domain adaptation, respectively. Note that the performance of normalization-basel methods the single (Table [T] and in the continu- ously changing (Table 2 settings are identical. Tables 4 and show the adaptation performance on the source and class imbalanced target domains, respectively. More results and discussions are provided in the appendix [ importantly, including results ImageNet-C (B.5).
Robustness to practical settings. In Table and 3 TTN and TTN applied methods show robust performance over the test batch size ranges from 200 to 1. Comparing with cormalization-basse baselines, we demonstrate that TTN, which uses channel-wisely wisely optimized combining rate a, shows better results than defining as constant hyperparameter, which can be considered special
6

---

Published as a conference paper at ICLR 2023
Table 3: Mixed domain adaptation on corruption benchmark. Error rate (L) of mixed domain with severity level 5 using WideResNeet44-22 as backbone for each test batch size. We used the reported results of MixNorm with fixed parameters from the original paper and denoted them as *.
CIFAR-10-C
CIFAR-100-C
Method
200
64
16
4
Avg.
200
64
16
Avg.
Source (CBN)
18.27
18.27
18.27
18.27
18.27
18.27
18.27
46.75
46.75
46.75
46.75
46.75
46.75
46.75
TBN
14.99
15.29
17.38
26.65
35.59
90.00
33.31
39.88
40.48
43.73
59.11
80.30
98.91
60.40
Norm
AdaptiveBN
12.62
12.48
12.97
14.59
15.74
16.02
14.07
36.88
36.86
38.49
41.43
43.38
44.31
40.23
a-BN
13.78
13.78
13.99
14.61
15.07
15.20
14.41
40.25
40.11
40.47
41.64
42.39
43.81
41,45
MixNorm' r
18.80
18.80
18.80
18.80
18.80
18.80
18.80
Ours (TTN)
12.16
12.19
12.34
13.96
15.55
17.83
14,00
36.24
36.23
36.85
41.01
45.85
55.52
41.95
TENT
14.33
14.97
17.30
26.07
35.37
90.00
33.01
39.36
40.01
43.33
58.98
80.55
98.92
60.
Optim.
+Ours (TTN)
12.02
12.04
12.20
13.77
15.42
16.40
13.64
36.29
36.23
36.89
41.38
46.65
57.95
42.57
SWR
13.24
13.06
16.57
26.08
38.65
91.03
59.54
37.84
37.93
44.37
59.50
78.66
98.95
33.10
+Our (TTN)
11.89
11.65
13.37
17.05
23.50
64.10
50.29
36.49
36.51 39.60
46.20
58.20 84.76
23.59
case of TTN; TBN and a-BN corresponds to a 1 and 0.1, respectively. More comparisons with
different constant a are provided in the appendix B.2 It is noteworthy that TTN as a stand-alone method favorably compares with optimization based baselines in all three scenarios.
Table 4: Source domain adaptation. Error rate Table 5: Class imbalanced target domain. Error
(L) on CIFAR-10 using NedeResNeet40-22
rate (J) averaged over 15 corruptions of CIFAR-
10-C with severity level 5 using WedeResNet-40-
Method
Test batch size
Avg.
2. Details are provided in the appendix A.2
Source (CBN)
4.92
4.92
4.92
4.92
4.92
4.92
4.92
Oycim. Norm
TBN
6.41
6.60
8.64
17.65
26.00
06.08
50.00
25,00
Method
Test
Ang.
Ours (TTN)
4.88
5.11
5.35
7,27
9.45
9.96
7,00
200
+Ours (TIN)
TENT
6.15
4.9
6.45
5.11
5.32
K61
17.61
7.23
26.20
9.38
90.00
10.21
7,02
32.2
Source (CBBN
18.27
18.27
18.27
18.27
18.27
18.27
18.27
+Owes (TTN)
SWR
5.65
4.79
601
502
5.1
K25
17.49
6.68
26.32
7.91
90.00
9.34
25.62
6.54
TBN
(TTN)
35.75
77.60
76.66
34.92
77.52
78.59
28.60
77.84
90.00
17.09
30,82
79.74
Adopting TTN improves other TTA methods. We compare optimization-baser methods with and without TTN layers. Since TENT, SWR, and COTTA optimize model parameters on top of using TBN layers, they also suffer from performance drops when the test batch size becomes small. Adopting TTN reduces the dependency on large test batch size, i.e. makes robust to small batch size, and even improves their performance when using large test batch. Furthermore, in continual 3 adaptation scenario, TENT and SWR shows higher error rate than in single domain (Table I adaptation. We interpret that because they update the model parameters based on the current output and predict the next input batch using the updated model, the model will not perform well if the consecutive batches have different corruption types (i.e., mixed domain adaptation). Moreover, the error from the previous input batch propagates to the future input stream, and thus they may fall apart rapidly once they have a strongly wrong signal, which can happen in continual adaptation (i.e., long-teni -erm adaptation without resetting). Applying TTN significantly accelerates their model performance regardless of the evaluation scenarios.
(Table 2 and mixed domain (Table
TTN preserves knowledge on source domain. In practice, data driven from the source domain (or a merely different domain) can be re-encountered in test time. We used clean domain test data in the single domain adaptation scenario show how TTN and other TTA methods adapt to the seen source domain data (but unseen instance). As shown in Table 4. all baseline methods using TBN layers, show performance drops even with large batch sizes. We can conclude that it is still better to rely on source statistics collected from the large training data than using only current input statistics, even if its batch size is large enough to obtain reliable statisties (i.e., 200). However, since TTN utilizes source statistics while leveraging the current input, TTN itself and TTN adopted methods well preserve the source knowledge compared to the TBN-based methods. With a batch size of 200, we observe that combining the source and current test batch statistics outperforms the source model (see 3rd row of Table 4).
TTN is robust to class imbalanced scenario. Heavily depending on current test batch statistics are especially vulnerable when the class labels are imbalanced (Boudiaf et al. 2022 Gong et al.) 2022). To simulate this situation, we sorted test images in class label order and then sampled test batches following the sorted data order. In Table 5 we observe that TTN is more robust to the class imbalanced scenario than utilizing only test batch statistics (i.e., TBN), As explained in Section 3.5
7

---

Published as a conference paper at ICLR 2023
Table 6: Adaptation on DG benchmarks in semantic segmentation. mloU(() on four unseen domains with test batch size of 2 using ResNet-50 based DeepLabV3+ as backbone.
Method (Cityscapes
+)
BDD-100K
Mapillary
GTAV
SYNTHIA
| Cityscupes
Source Chen et
2018
43.50
54.37
43.71
22.78
76.15
Norm
TBN
43.12
47.61
42.51
25.71
72.94
Ours (TTN)
47.40
56.92
44.71
26.68
75.09
TENT
43.30
47,80
43.57
25.92
72.93
Optim.
Ours (TTN)
47.89
57.84
46,18
27.29
75.04
SWR
43.40
47.95
42.88
25.97
72.93
Ours (TTN)
48.85
59.09
46.71
29.16
74.89
we are putting more importance on CBN than TBN, where semantic information is mainly handled, understand that TTN is less impacted by skewed label distribution.
Le.. in deeper layers,
3.3 EXPERIMENTS ON SEMANTIC SEGMENTATION
We additionally conduct experiments on domain generalization (DG) benchmarks (Choi et al, 2021 Pan et al.) 2018 for semantic segmentation. including natural domain shifts (e.g., Mlyscapes-BBDD 100K), to demonstrate the broad applicability of TTN. Table/Jshows the results of evaluating the ResNet--5--based Dae+ (Chen et al. 2018) model trained on the Cityscapes training set using the validation set of real-world datasets such as Cityscapes, BDD-100K, and Map- illary, and synthetic datasets including GTAV and SYNTHIA. We employ a test batch size of 2 for test-time adaptation in semantic segmentation. We observe that even when exploiting test batch statistics for standardization in BN layers (TBN) or updating the model parameters on top of TBN (TENT, SWR) does not improve the model performance (i.e.. perform worse than the source model), adopting TTN helps the model make good use of the strength of the test batch statistics. Implemen- tation details and additional results are provided in the appendix A. and B.7 respectively.
3.4 ABLATION STUDIES
Prior A regularizes a to be robust to overall test batch sizes. We conduct an ablation study on the importance of each proposed component, i.e., initializing a with prior A, optimizing a using CE and MSE losses, and the results are shown in Table 7 Using A for initialization and MSE loss aims to optimize a following our intuition that we discussed in Section 2.3 Optimizing a using CE loss improves the overall performance, but without regularizing with MSE loss, a may overfit to large batch size (rows
3). Initialization with A or not does not show a significant difference, but A provides a better starting point than random initialization when comparing the left and right of the 2nd row. We observe that when using MSE loss, regardless of initialization using A, the optimized sufficiently reflects our intuition resulting in low error rate to overall batch sizes (row 3).
Table 7: Ablation study on importance of each component
Method
Test batch size
Method
Test batch size
Avg.
Avg.
Init.
MSE
200
64
16
Init. MSE 200 64
16
/
13.36
13.43
13.85
15.50
17.43
20.07
15.61
13.37 13.43
13.85
15.50
17.44
20.07
15.61
11.64
11.73
12.26
14.46
16.94
19.88
14.49
/
11.73
11.82
12.23
14.18
16.41
19.27
14.27
11.78
12.21
13.97
15.86
18.00
13.91
/ 11.80
12.13
13,93
15.83
17.99
13.89
3.5 VISUALIZATION OF a
Fig. shows the visualization of optimized a for CIFAR-10 using WideResNet-40-2. We observe that decreases from shallow to deep layers (left to right), which means CBN is more active in deeper layers, and TBN is vice versa. As shown in Table 4 and 6 CBN employing source statistics is superior to TBN when the distribution shift between source and target domains is small. Assum- ing that the a we obtained is optimal, we can conjecture that CBN is more active (i.e., a closer to 0) in deeper layers because domain information causing the distribution shift has been diminished. In contrast, TBN has a larger weight (i.e., a closer to 1) in shallower layers since the domain in- formation induces large distribution shift. This interpretation is consistent with the observations of previous studies (Pan et al.] 2018 Wang et al.] |202 Kim et al.) 2022a) that style information mainly exists in shallower layers, whereas only content information remains deeper layers.
8

---

Published as a conference paper at ICLR 2023
0.9
Channel-
0.8
Channel mean per layer
0.5
044
0.3
0.2
Channels in all layers
2661
Figure 4: Optimized o. and y-axes indicate all channels in order from shallow deep layers and the interpolating weight a, respectively. C₁ denotes the channel size of layer I.
RELATED WORK
Test-time adaptation/training (TTA) aims to adapt models towards test data to overcome the per- formance degradation caused by distribution shifts (Sun et al.
2020 Wang al. (2020). There are
other related problems, unsupervised domain adaptation (UDA) (Sun & Saenko 2016,]
(0066,Gaani Ganin et al.
2016) and source-free domain adaptation (SFDA) (Liang et al. 2020 2020 [uang et al. 2021] Liu al.
2021]. Both UDA and SFDA have access to sufficiently large enough unlabeled target datasets, and their objective is to achieve high performance on that particular target domain. Unlike UDA and SFDA, TTA utilizes test data in an online manner. There are two key factors of recent approaches: adapting standardization statistics in normalization layers and adapting model parameters.
Normalization in Test Time. Nado et al. (2020) suggested prediction-time BN, which uses test batch statistics for standardization and Schneider et al. (2020) introduced to adapt BN statistics by combining source and test batch statistics considering the the test batch size to mitigate the inter- mediate covariate shift. this paper, we refer to the former method as TBN. Similarly, You et al. (02I and Khurana et al. (202T) mixed the statistics using predefined hyperparameter. Also, MMiza et al. (022) and et al. 2 adapted the statistics using moving average while augmenting the input to form a pseudo test batch when only single instance is given. The primary difference with the existing approaches is that we consider the channel-wist domain-shift sensitivity of BN layers optimize the interpolating weights between CBN and TBN. Concurrently, Zou al. (2022) pro- posed to adjust the standardization statistics using a learnable calibration strength and showed its effectiveness focusing on the semantic segmentation task.
Optimization in Test Time. TENT (Wang et al. 2020). SWR (Choictal 2022) and CoTTA (Wang
eet al., 2022) updated model parameters while using TBN. TENT optimized affine parameters in BN layers using entropy minimization while freezing the others. To maximize the adaptability, SWR updated the entire model parameters minimizing the entropy loss based on the domain-shift sensitivity. To stabilize the adaptation in continuously changing domains, CoTTA used consistency loss between student and teacher models and stochastically restored random parts the pre-trained model. Liu et al. (2021) and Chen et al. 2022) suggested to update the model through contrastive learning.
We focus on correcting the standardization statistics using domain-shiff aware interpolating weight a. Similar to (hho et al.] 2022), we measure the domain-shift sensitivity by comparing gradients. The principle difference that channel-wise sensitivity when optimizing a in post-training, while SWR used layer-wise sensitivity regularizing the entire model update in test time.
CONCLUSIONS
This paper proposes TTN, novel domain-shift aware batch normalization layer, which combines the benefits of CBN and TBN that have a trade-off relationship. We present strategy for mixing CBN and TBN based on the interpolating weight derived from the optimization procedure utilizing the sensitivity to domain shift and show that our method significantly outperforms other normaliza- tion techniques in various realistic evaluation settings. Additionally, our method is highly practical because it can complement other optimization-
TTA methods. The oracle mixing ratio between CBN and TBN can vary depending on the domain gap difference. However, our proposed method employs a fixed mixing ratio during test time, where the mixing ratio is optimized before model denloyment. If we could find the optimal mixing ratio according to the distribution shift during test time, we can expect further performance improvement. We consider it as future work. In this regard, efforts encourage this field to become more practical and inspire new lines of research.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our method, we provide the experimental setup in Section 3.1 Moreover, the details on implementation and evaluation settings can be found in the appendix and A.2 respectively. The pseudocode for overall training and testing scheme is provided in the appendix A.3 Together with related references and publicly available codes, we believe our paper contains sufficient information for reimplementation.
ACKNOWLEDGEMENT
We would like thank Kyuwoong Hwang, Simyung Chang, and Seunghan Yang of the Qualcomm AI Research team for their valuable discussions.
REFERENCES
Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-free online test-time adaptation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
2022.
John Bronskill, Jonathan Gordon, James Requeima, Sebastian Nowozin, and Richard Turner. Tas- knorm: Rethinking batch normalization for meta-learning. In International Conference Ma- chine Learning (ICML). PMLR, 2020.
Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
Liang-Chieh Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference computer vision (ECCV), 2018.
Liang-
Sungha Choi, Sanghun Jung, Huiwon Yun, Joanne Kim, Seungryong Kim, and Jaegul Choo. Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2021.
Sungha Choi, Seunghan Yang, Seokeon Choi, and Sungrack Yun. Improving test-time adaptation shift-agnostic weight regularization and nearest source prototypes. In European Conference on Computer Vision (ECCV), 2022.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Nomain-adverseee training of neural net- works. The journal of machine learning research, 2016.
Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. Noterrobust continual test-time adaptation against temporal correlation. Advances Neural In- formation Processing Systems (NeurIPS), 2022.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint (tti:::706006777 2017.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Ad- vances Neural Information Processing Systems (NeurIPS), 2004.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
10

---

Published as a conference paper at ICLR 2023
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. In International Conference on Learning Representations (ICLR), 2018.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. In Interaatinnal Conference on Learning Representations (ICLR), 2019.
Xuefeng Hu, Gokhan Uzunbas, Sirius Chen, Rui Wang, Ashish Shah, Ram Nevatia, and Ser-Nam Lim. Mixnorm: Test-time adaptation through online normalization estimation. arXiv preprint 1rXiv:2110.11444 2021.
Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data. 2021.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), 2015.
Ansh Khurana, Sujoy Paul, Piyush Rai, Soma Biswas, and Gaurav Aggarwal. Sita: Single image test-time adaptation. arXiv preprint arXiv:2112.0233: 2021.
Byeonggeun Kim, Seunghan Yang. Jangho Kim, Hyunsin Park, Juntae Lee, and Simyung Chang- Domain Generalization with Relaxed Instance Frequency-wise Normalization for Multi-device Acoustic Scene Classification. In Conference of the International Speech Communication Asso- ciation (INTERSPEECH), 2022a.
Jin Kim, Jiyoung Lee, Jungin Park, Dongbo Min, and Kwanghoon Sohn. Pin the memory: Learn- ing to generalize semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022b.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2015.
Suhyeon Lee, Hongje Seong, Seongwon Lee, and Euntai Kim. Wildnet: Learning domain general- ized semantic segmentation from the wild. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Jian Liang Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference Machine Learning (ICML). PMLR, 2020.
Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexan- dre Alahi. Ttt++: When does self-supervised test-time training fail or thrive? Advances Neural Information Processing Systems (NeurIPS), 2021.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Unsupervised domain adap- tation with residual transfer networks. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. 2017.
M Jehanzeb Mirza, Jakub Micorek, Horst Possegger, and Horst Bischof. The norm must go Dynamic unsupervised domain adaptation by normalization. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Zachary Nado, Shreyas Padhy, Sculley, Alexander D' A'mou, Balaji Lakshminarayanan, and Jasper Snock. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963, 2020.
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In International Conference Computer Vision (ICCV), 2017.
Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at once: Enhancing learning and generalization capacities via ibn-net. In European Conference Computer Vision (ECCV), 2018.
11

---

Published as a conference paper at ICLR 2023
Joaquin Quinonero- Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. Mit Press, 2008.
Stephan R. Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European Conference on Computer Vision (ECCV), 2016.
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Ad- vances in Neural Information Processing Systems (NeurIPS), 2020.
Saurabh Singh and Abhinav Shrivastava. Evalnorm: Estimating batch normalization statistics for evaluation. In International Conference on Computer Vision (ICCV), 2019.
Cecilia Summers and Michael Dinneen. Four things everyone should know to improve batch normalization. In International Conference on Learning Representations (ICLR), 2019.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European Conference on Computer Vision (ECCV), 2016.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time train- ing with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning (ICML), 2020.
Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Adver- sarial entropy minimization for domain adaptation in semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Dequann Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Repre- sentations (ICLR), 2020.
Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Yulin Wang, Zanlin Ni, Shiji Song, Le Yang, and Gao Huang. Revisiting locally supervised learning: an alternative to end-to-end training. International Conference on Learning Representations (ICLR), 2021.
Fuming You, Jingjing Li, and Zhou Zhao. Test-time batch statistics calibration for covariate shift. arXiv preprint arXiv:21 10.00065. 2021.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madha- van, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Yuliang Zou, Zizhao Zhang, Chun-Liang Li, Han Zhang, Tomas Pfister, and Jia-Bin Huang. Learn- ing instance-specific adaptation for cross-domain segmentation. European Conference on Com- puter Vision (ECCV), 2022.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX: FOR REPRODUCIBILITY
This section provides supplemental material for Section 2 and 3.1
A.1 IMPLEMENTATION DETAILS
Datasets and Models. For SEFAR-10/100-2 we optimized a using augmented CIFAR- 10/100 training set on the pre-train pre-trained WideResNet-40-22 (WRN-40) (Hendrycks et 2019). For ImageNet- C, we used augmented ImageNet training set (randomly sampled 64000 instances per epoch) on the pre-trained ResNet-50.
Augmentation. Following the setting of SWR (Choi et al.) 2022), we used color jittering, random invert and random grayscale when obtaining the prior A. When optimizing a, we followed the augmentation choice of CoTTA <Wang et al. 2022). which are color jittering, padding, random affine, gaussian blur, center crop and random horizontal flip. We excluded for gaussian noise to avoid any overlap with corruption type of common corruptions (Hendrycks & Dietterich/ For ImageNet, we only used the same augmentation both for obtaining A and optimizing a following the SWR augmentation choices.
Post-training. When obtaining prior, we used randomly selected 1024 samples from the training set, following the setting of SWR. We used Adam (Kingma & Ba 2015) optimizer using learning rate (LR) of le-3, which is decayed with cosine schedule (Loshchilov & Hutter 2017) for 30 epochs and used 200 training batch for CIFAR-10/100 For ImageNet, we lowered LR to 2.5e-4 and used 64 batch size. For semantic segmentation task, we trained TTN using Cityscapes training set, and training batch size of 2. We resized the image height to 800 while preserving the original aspect ratio Cordts et al.] (2016). We can terminate the training when MSE loss saturates (We observed that a does not show significant difference after the MSE loss is saturated). We used the weighting hyperparmeter to MSE loss À as 1.
Test time. For AdaptiveBN, which adjusts the interpolating weight using two factors: hyperpa- rameter N and test batch size n, we followed the suggested N, which is empirically obtained best hyperparameter from the original paper, for each n (Figure 1 11, ResNet architecture from Schneider et al. (2220)) In detail, we set N as 256, 128, 64, 32, and 16 for test batch size 200, 64, 16, 4, 2, and which yields a as 0.44, 0.33, 0.2, 0.11, 0.06, and 0.06, respectively.
For optimization-baseee TTA methods, we followed default setting in TENT, SWR, and CoTTA for test-time adaptation. We used LR of 1e-3 test batch size of 200 for CIFAR-10/100-C in single domain (TENT, SWR) and continuously changing domain (CoTTA) scenarios. To avoid rapid error accumulation, we lowered LR to le-4 for TENT and SWR in continual and mixed do- main scenarios. Moreover, we updated model parameters after accumulating the gradients for 200 samples for CIFAR-10/100-C. In other words, we compute gradients per batch, but update, i.e., optimizer: step (), after seeing 200 data samples. Exceptionally, we used LR of 5e-5 for SWR and SWR+TTN in mixed domain setting. Additionally, in continuously changing and mixed domain scenarios, used the stable version of SWR, which updates the model parameter based on the frozen source parameters instead of previously updated parameters (original SWR).
For semantic segmentation, we set the test batch size and learning rate for optimization- methods as le-6 for all datasets. For SWR, we set the importance of the regularization term A, as 500. The other hyperparameters are kept the same as Choi al. (2022) choices. For all test-time adaptation, we used constant learning rate without scheduling.
A.2 EVALUATION SCENARIO DETAILS
Class imbalanced setting. In the main paper Table we show the results under class imbalanced settings. In the setting, we sorted the test dataset of each corruption in the order of class labels, ie., from class 0 to 9 for CIFAR-1O-C. Then, we comprised test batches following the sorted order. Therefore, most batches consist of single class input data, which leads to biased test batch statisties. For larger batch size, the statistics are more likely to be extremely skewed, and that explains why emror rates are higher with larger batch sizes than with the small ones.
13

---

Published as a conference paper at ICLR 2023
A.3 PSEUDOCODE
Pseudocode for post-training, i.e., obtaining A and optimizing a, provided in Algorithms and 2 respectively, and that for test time in Algorithm 3 Moreover, we provide HyTocch-frendde pseudocode for obtaining A in Listing [ Please see Section 2 for equations and terminologies used the algorithms.
Algorithm Obtain prior A
Require: Pre-trained model fe: source training data Ds (X
2: Output: Prior A
3: for all (x, y) in D.s do
4: Augment x:: x²'
5: Collect gradients (V V and (V3, VBr) from fo using clean x and augmented
6: end for
7: Compute gradient distance score using Eq.
8: Define prior A using Eq. [5]
Algorithm Post-train
1: Require: Pre-trained model fe: source training data Ds = (X, Y); step size hyperparameter 77: regularization weight hyperparameter À
2: Output: Optimized interpolating weight a
3: Obtain prior A using Algorithm
4: Initialize a with prior A
5: Replace all BN layers of So TTN layers using and Eq. 2
6: while not done do
7: Sample minibatches 8 from Ds
8: for all minibatches do
9:
Augment all æ in 1S: x²
Evaluate V. using fo given ((1). (1) 15" using Eq. 6 while adapting standardization
10:
statistics using Eq. 2
Update < v.
12: end for
end while
Algorithm 3 Inference (Test time)
1: Require: Pre-trained model fo: optimized target test data Dt- (X);
2: Replace all BN layers of fe to TTN layers using a and Eq. 2
3: Sample minibatches Bd from Dr
4: for all minibatches do
5: Make prediction using So given B" while adapting standardization statisties using Eq.
6: end for
def obtain prior (model, train data):
2 # make weight and bias of BN layers requires_grad-Truee otherwise False
params = (n: for n, p in model. nww.d parameters () if p. requires_graad
grad_sim ()
for x, y in train_data: # collect gradients for clean and augmented input
grad_ org = collect_ grad (model, y, params) grad_au aug collect_grad (model, augment (x), Y, params)
10
11
compute grad similarity
12
for n, in params... itemssttmmss (): [rad_sim[n]] .ataa cosine_50 in grad_aug)
14

---

Published as a conference paper at ICLR 2023
average over data samples
16
for n, P params. items (): grad_sim[n] data /= len (train_data)
17
18
# min max normalization
max_grad get_max_valle value ggrad_sim) scalar
21
min_grad get get_min_value (grad_sim)) scalar
22
grad_aist = []
23
for n, p in staad_sim.itss (): graa dist|n|
min grad) (max grad min_grad)
25
26
prior - []
27
j -
# integrate gradients of weight (garma) and bias (beta) for each BN Layer
for n, P grad dist.items ()
30
"weight" n:
31
arper.append())
32
clif "bias" in
priorlj] +
prier[i] /- prior[j] []]) j +=1
35
36
37
return prior
39
40
def col collect_grad (model, x, y, params)
41
model. zereggrad))))
model (x)
loss 44
loss (out, y)
loss. loss.backward()
45
46
grad (}
for n, p in params items ()
48
grad(n) data p.data
49
50
return grad
Listing 1: Py Torch--fiendll pseudo code for obtaining prior
B APPENDIX: EXPERIMENTAL RESULTS
B. ABLATION STUDIES
Channel-wise a. In Table we compared different granularity levels interpolating weight a, i.e., channel-wise layer-wise, and constant value with CIFAR-10-C and backbone WRN-40, We ob- served that channel-wise optimized a shows the best performance. We take average the channel- wisely optimized a (the 1st row) over channels to make layer-wise (the 2nd row), and take average over all channels and layers to make a constant value (the 3rd row). a of the 1st and 2nd are visualized in the main paper Figure 4 colored in blue and red, respectively. The constant value a (the 3rd row) is 0.3988. We also optimized a layer-wisely (the 4th row).
Table 8: Ablation study granularity level of a
Test batch size
Method
Avg.
200
64
16
1
Channel-wise (Optimized)
11.67
11.80
12.13
13.93
15.83
17.99
13.89
2
Layer-wise (Channel mean)
12.75
12.84
13.16
14.66
16.40
18.82
14.77
3
Constant (Mean)
12.07
12.21
13.05
16.72
20.04
21.26
15.89
Layer-wise (Optimized)
13.11
13.21
13.51
14.84
16.46
18.62
14.96
15

---

Published as a conference paper at ICLR 2023
MSE loss strength X. We empirically set the MSE loss strength À in Eq. as through the ablation study using CIFAR-IO-C and WRN-40 (Table 9p. Using the MSE regularizer prevents the learnable from moving too far from the prior, thus letting follow intuition, ie.. putting smaller im- portance on the test batch statistics if the layer or channel is invariant to domain shifts. However, with too strong regularization (A=10.0), the overall error rates high, which means the a needs to be sufficiently optimized. On the other hand, with too small regularization, the a may overfit to the training batch size (B=200) and lose the generalizability to the smaller batch size.
Table 9: MSE loss strength À
Test batch size
A
Avg.
200
64
0.0
11.73
11.82
12.23
14.18
16.41
19.27
14.27
0.1
11.65
11.91
12.09
13.84
15.71
18.24
13.91
1.0
11.67
11.80
12.13
13.93
15.83
17.99
13.89
10.0
12.45
12.63
12.82
14.55
16.32
18.56
14.56
B.2 MORE COMPARISONS ON CIFARIO-C
Constant a. Table 10 shows results of simple baseline for normalization-basee methods where the is a constant value ranging from 0 to
0 identical CBN (loffe & Szegedy, 2013). and a 1 is identical to TBN Nado et al. 2020). We observe that the lower a, i.c., using less test batch statistics, shows better performance small test batch sizes. This observation is analogous the
finding of the previous work (Schneider et al. 2020).
Table 10: Constant a. Error rate (L) averaged over 15 corruptions of CIFAR- (WRN-40).
Test batch size
Avg.
200
64
16
4
2
0
18.26
18.39
18.26
18.26
18.25
18.25
18.
0.1
13.95
14.1
14.05
14.65
15.14
15.45
14.56
0.2
12.46
12.66
12.89
14.30
15.53
15.64
13.91
0.3
12.05
12.29
12.72
15.18
17.35
17.42
14.50
0.4
12.13
12.41
13.12
16.69
19.81
20.51
15.78
0.5
12.42
12.78
13.73
18.32
22.52
24.88
17.44
0.6
12.88
13.32
14.48
20.02
25.17
31.97
19.64
0.7
13.37
13.9
15.23
21.75
27.91
46.65
23.14
0.8
13.82
14.37
15.94
23.44
30.59
77.15
29.22
0.9
14.18
14.8
16.58
24.94
33.12
89.81
32.24
1
14.50
15.15
17.10
26.29
35.67
90.00
33.12
B.3 VARIANTS OF TTN FOR SMALL TEST BATCH SIZE
Online TTN. TTN interpolating weight a can also be adapted during test time. Table II shows the results of the TTN online version, which further optimizes the post-trained alpha using the entropy minimization and mean-squared error (MSE) loss between the updated alpha and the initial post- trained alpha. We followed entropy minimization loss used in TENT (Wang al. 2020). and the MSE loss can be written as LMSR = ||a= - oo where ogg is the post-trained a. We set the learning rate 1e-2, 2.5e-3, 5e-4, le-4, 5e-5, and 2.5e-5 by linearly decreasing according to the test batch size of 200, 64, 16, 4, 2, and 1 (Goyal et al.) 2017). The online TTN shows improvements compared the offline TTN in all three evaluation scenarios (single, continuously changing, and mixed).
Scaled TTN. Following the observation from Table 10 we lowered the interpolating weight by mul- tiplying a constant scale value, ranging (0, 1), the optimized TTN a. In Table I1 set the scale value as 0.4.
we empirically
Dynamic training batch size. We observe that using the dynamic batch size in the post-training stage also improves the performance for small test batch sizes (2 or 1), while slightly deteriorating the performance for large test batch sizes (200 or 64). We randomly sampled training batch size from the range of [4, 200] for each iteration. Other hyperparameters are kept as the same.
16

---

Published as a conference paper at ICLR 2023
Table 11: TTN variants. Error rate (4) averaged over 15 corruptions of CIFAR-IO-C (WRN-40).
Test batch size
Avg.
Method
Eval. setting
||
200
64
16
4
2
TTN (offfine, default)
Single & Cont.
11.67
11.80
12.13
13.93
15.83
17.99
13.89
TTN (offline, default)
Mixed
12.16
12.19
12.34
13.96
15.55
17.83
14.00
TTN (online)
Single Cont.
11.39
11.64
11.97
13.70
15.41
17.49
13.60
TTN (online)
11,67
11.96
12.15
13.90
15.67
17.72
13.85
TTN (online)
Mixed
12.04
12.04
12.06
13.90
15.46
17.62
13.85
TTN (scaled)
Single & Cont.
13.20
13.38
13.35
13.88
14.54
15.17
13.92
TTN (sculed)
Mixed
13.17
13.05
13.17
13.74
14.36
15.09
13.76
TTN (dynamic train batch size)
Single & Cont.
11.82
12.04
12.17
13.60
15.13
17.22
13.66
TTN (dynamic train batch size)
Mixed
12.12
11.91
13.43
14.76
17.23
13.58
B.4 STUDY ON AUGMENTATION TYPE
TTN is robust to the data augmentation. We used data augmentation in the post-training phase to simulate domain shifts. The rationale for the simulation is to expose the model to different input domains. Especially when obtaining the prior, we compare the outputs from shifted domains with the clean (original) domain in order to analyze which part of the model is affected by the domain discrepancy. Therefore, changing the domain itself is what matters, not which domain the input is shifted to. We demonstrated this by conducting ablation studies by varying the augmentation type.
We analyzed various augmentation types when obtaining prior and when optimizing alpha and the results are shown in Figure 5 (a) and (b), respectively. We use a true corruption, i.e., one of 15 corruption types in the corruption benchmark, to analyze how TTN works the augmentation type and test corruption type are misaligned or perfectly aligned. Specifically, the augmentation choices when optimizing alpha as described in the appendix [..I, and vice versa. Expectedly, the diagonal elements, i.e., where the same corruption type is used both for in post- training and in test time, tend to show the lowest eπror rate in Figure b). The row-wise standard deviation lower than prior is invariant to the augmentation choice. Moreover, we observe that the average error rate over all elements, 11.74% in ablation for obtaining prior and 11.67% % in ablation for optimizing alpha, is almost same TTN result 11.67% (See Table 14 and Figure 5). Moreover, we conducted an ablation study on the choice of the augmentation type using CIFAR- and WRN-40 (Table[2). We observe that obtaining prior and optimizing alpha steps are robust to the augmentation types.
augmentation type in the post-training phase
used the true corruption when obtaining the prior while keeping
most cases and even lower than 0.5 in Figure aa),, which means the
Table 12: Ablation study on augmentation choice. From left to right one augmentation type is added at a time. Default setting, which we used in all experiments, is colored in gray.
color jitter
grayscale
invert
+ gaussian blur
+ horizontal flip
Prior augmentation type
12.03
11.83
11.67
11.59
11.58
color jitter
+ padding
+ affine
+ gaussian blur
horizontal flip
Optimizing augmentation type
11.78
11.78
11.7
11.70
11.67
B.5 RESULTS ON IMAGENET-C
Table 13 shows the experimental results using ResNet-50 (He et al.) 2016) on ImageNet-
C (Hendrycks & Dietterich 2018) dataset. We emphasized the effectiveness our proposed method by showing the significant improvement in large-scale dataset experiment. Similar to the results in
CIFAR-10/100-C; TTN showed the best performance compared to normalization- based methods (TBN (Ndo et al., 2020), AdaptiveBN (Schneider et al. (2020), and a-BN (You et al.) (221)) and improved TTA performance when it is applied to optimization- based methods (TENT (Wang et al.)
2020 and SWR (Choi et al. 2022)
During post-training we used LR of 2.5e-4 and batch size of 64. In test time, we used the learning rate of 2.5c-4 for TENT following the setting from the original paper, and used 5e-6 for TENT+TTN. For SWR and SWR+TTN, we used learning rate of 2.5e-4 for B=64, and 2.5c-6 for B=16, 4, 2, and 1. We had carefully tune the learning rate especially for SWR, since the method updates the
17

---

Published as a conference paper at ICLR 2023
entire model parameters in unsupervised manner and hence is very sensitive to the learning rate when the test batch size becomes small. Other hyperparameters and details are kept same (See the appendix A.1 for more details). Moreover, to avoid rapid accumulation, sufficiently large test samples and then updated the model parameters (for example, we conducted adaptation in every 64 iterations in the of batch size of 1) in both TENT and SWR.
stored gradients for
Table 13: Single domain adaptation on ImageNet-C (ResNet-50). Error rate (L) averaged over 15 corruption types with severity level 5 is reported for each test batch size.
Test batch size
Method
Avg. Error
64
16
Sourec (CBN)
93.34
93.34
93.34
93.34
93.34
93.34
TBN
74.24
76.81
85.74
95.35
99.86
86.40
Norm
Bd
77.86
81.47
86.71
90.15
91.11
85,46
a-BN
86.06
86.32
87.16
88.33
90.45
87.66
Ours (TTN)
72.21
73.18
76.98
81.52
88.49
78.48
TENT
66,56
72.61
93.37
99.46
99,90
86.41
Optim.
+Ours (TTN)
71.42
72.45
76.66
81.89
91.00
78.68
SWR
64.41
74.19
84.30
93.05
99.86
83.16
+Ours (TTN)
55.68
78.48
86.37
94.08
76.77
B.6 ERROR RATES OF EACH CORRUPTION
In Table 14
we show the error rates of TTN for each corruption of CIFAR-IO-C using the WRN-40
backbone. The averaged results over the corruptions are in Table [
Table 14: Results of each corruption (CIFAR-10-C)
hatch
shoc
impulse
defocus
brightness
pisel.
jpsg.
Avg.
12.78
1.23
11.29
12.81
7.41
7.41
15.23
13.00
17.98
7.71
18:46
8.95
7.68
10.85
10.17
12:21
6.25
8.54
13.95
15.67
15.34
12:13
17.03
15.
19.75
9:26
20.93
10.01
9.62
12.58
11:85
13.65
7.69
9.25
16.21
18.08
17.70
13:03
19.21
16.80
21.86
10.70
23.74
11.39
11.92
14,00
13.39
15.38
3.95
10.13
18.92
20.68
20.40
15.83
22.03
19.97
25.34
12.41
36.99
12.22
14.39
14.73
14.00
17.27
10.16
9.51
22.10
24.12
24.09
17.99
B.7 SEGMENTATION RESULTS
For more comparisons, we add segmentation results for TBN and TTN using test batch size (B) of 4 and 8 (Table The results demonstrate that TTN is robust to the test batch sizes. In other words, the performance difference across the test batch size is small when using TTN (TTN with B=2, 4, and 8). The results are averaged over 3 runs (i.e., using 3 independently optimized a). and the standard deviation denoted with a + sign. We omitted the standard deviation for TBN, which is 0.0 for every result since no optimization is required. Since the backbone network is trained with a train batch size of 8, we assume that test batch statistics estimated from the test batch with B=8 sufficiently reliable. Accordingly, TBN with B=8 shows compatible results. However, when B becomes small (i.c., in more practical scenario), problematic test batch statistics are estimated, and thus TBN suffers from the performance drop while TTN keeps showing robust performance. It is worth noting that TTN outperforms TBN by 3.77% in average accuracy when B=2, i.e., in the most practical evaluation setting, and by 2.54% and 1.99% for B=4 and 8, respectively.
Table 15: Adaptation DG benchmarks in semantic segmentation. mloU(t) four unseen domains with test batch size of 2, 4, and 8 using ResNet-50 based DecpLab V3+ as backbone.
Method (Cityscapes-+)
BDD-100K
Mapillary
GTAV
SYNTHIA
Cityscapes
TBN (B=2)
43.12
47.61
42.51
25.71
72.94
Ours (TTN) (B=2)
47.40 0.02)
56.88( 0.04)
44.69( 0.03)
26.68( 0.01)
75.0% 0.01)
Norn
TEN (B=4)
45.64
49.17
44.26
25.96
74.29
Ours (TTN) (B=4)
47.72( 0.01)
57.11 0.01)
45.08( 0.02)
26.52(+ 0.01)
75.56( 0.01)
TBN (B=8)
46.42
49,38
44.81
25.97
75.42
Ours (TTN) (B=8)
47.25 0.02)
57.28(4: 0.02)
45.13 0.03)
26.45( 0.01)
75.82 0.01)
18

---

Published as a conference paper at ICLR 2023
Error rate. Colored by norma lized value (across test corruption type)
a0g:
34.0
34.5
14.5
13.3
24.4
13:2
13:2
13.1
23.0
13.1
2b.2
34.5
13.2
24
15.
15.08
0.34
snot
12.7
12.3
15.1
12.7
12.9
13.7
17.6
134
12.1
12.8
12.79
0.22
irpube_ vobe
0.16
2.5
7.5
74
glss
17.0
17.8
17.0
31.1
37.8
16.2
18.0
10.1
17.9
17.8
18
17.6
12.9
37
18.0
17.91
0.14
motien Blur
0.0
5.4
5.4
8.4
8.4
0.4
0.08
Test coruppion type
200m, biur
10.4
10.3
10.3
30.0
20.3
20:5
30
10.4
30.4
10.3
30.0
10.2
10.5
20
30.5
10:44
0.14
8
best
9.9
9.6
3.9
10.
99
10
10
10
9-6
5.8
10.2
9.8
10
8.8
10
9.88
0.14
22.6
11.55
0.30
8.8
6.0
8.0
6.2
8.2
8333
8.3
9/2
81.3
8.0
6.7
11-4
01.3
9.6
8.3
12.
0.5
8.5
8.57
0.40
elest trannsttttoooooooooooom
13:4
13.4
13.2
13.3
13.3
13.3
13.4
13.2
13.3
13.34
0.07
1.59
reg_compressoe
15.0
13.7
18
15.0
10.7
16.7
16
ga goistaan notee
sht nolse
0e0011 our
glis bur
2009 DMH
1000
contrss
F0g cimoressionn
11.74
mpubee
Augmentation type used for obtaining prior
by normalized (across corruption type)
thot_ncise Locse
12.8
12.64
0.61
15.7
15.9
14.4
16.0
12.2
17.4
37.2
26.7
17.7
18.0
23.2
16.0
16.6
37.4
16.77
0.92
aetocus Bur
2.1
1.5
1.2
8.2
0.31
bloss_bur
U.4
8.3
5.6
0.5
8.76
0.29
Test corruption type
200m_bbur
7.4
7:3
7.4
2.4
7.3
2.0
2/4
2/7
2.3
7.9
2.5
2.3
7.0
X.
7.48
0.25
210W
10.2
10:4
20.4
10.2
10.5
9ost
12.9
12.6
17.4
13.0
15.2
11.0
15.0
13.4
12.40
1.45
trightness
0.4
0.4
0.0
0.09
contrast
33.2
13.2
23.4
11.3
13.3
24.2
11.3
13.35
0.37
pieelate
14C
14.4
14.1
24:2
14.0
13.7
13.8
14.1
14.5
13.0
13.9
25.4
15.1
10.6
14.9
14.58
1.27
jeg__come compeession
16.
14.5
24.0
14.5
14.8
14.8
14.9
15.1
15.4
34.1
24.4
14.3
0.39
0rtoss phr
2007 S0nL
200
K0
contraa bightmess
poelite
mebie
Augmentation type for optimizing alpha
Figure 5: True test corruption as augmentation. Each column represents the augmentation type used (a) when obtaining prior or (b) when optimizing a. Each row represents the test corruption type. The error rate ({) of CIFAR-10-CC with severity level and test batch size 200 using WRN-40 is annotated in each element. Element (i,j) represents the error rate when j-th corruption type used for augmenting the clean train data during post-training and tested i-th corruption test set. The heatmap is colored by error rate normalized across the test corruption type trow-wise normalization).
19


Reviews:
[{"review_id": "m50NUHj0EL", "reviewer": "Reviewer_RNFt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "**Summary:**\n\nThis paper presents a new test-time normalization (TTN) method that combines the training statistics and the test-time statistics via the importance between them. To obtain the Prior, the authors first augment the training samples as domain-shift ones, and then calculate the importance via the gradient distance score between the clean samples and domain-shift ones. The experimental results on CIFAR-C and ImageNet-C demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "I vote for reject since the proposed method has a strong assumption on the test samples, which may not be hold in practice.", "strength_and_weaknesses": "**Positive Points:**\n\n - The authors propose to adjust the balanced parameters between the training statistics and the test-time statistics based on the (augmented) training data.\n\n - The authors introduce a gradient distance score to calculate the balanced parameters $\\alpha$, which is technically sound.\n\n - The authors consider plenty of test-time scenarios, such as various batch sizes, stationary, continuously changing, and mixed domain adaptation.\n\n **Negative Points:**\n\n  - The authors obtain Prior based on the pair (clean sample x, and domain-shifted x'). And the domain-shifted x' greatly depends on the augmentation type and applying order. In this case, if the domain-shifted x' has an extensive domain gap from the test-time samples, the Prior would be estimated inaccurately, leading to poor TTA performance. Could the authors provide more discussion and explanations about this?\n\n - In the paragraph of \"Optimize $\\alpha$\", the authors state \"To simulate distribution shift, we use the augmented training data\". Here, the authors have a strong assumption that the test-time shifted type is very similar to the augmented type. However, in my opinion, this assumption often does not hold in practice. More explanations are required.\n \n - I suggest the authors can provide an ablation study on the augmentation type, which is very crucial for the proposed method. I wonder if there is an augmentations combination that works well for all the corruption-shifted test-time samples? If no, the proposed method may be difficult for real-world applications.\n\n - Most experimental results are reported on CIFAR-C. I suggest the authors can report the main results on ImageNet-C, which will be more convincing. I think the computational and time cost on ImageNet-C is similar to that on CIFAR-C. In Table 9, I find the authors do not report the results of \"Tent+TTN\" and \"SWR+TTN\". But I think these two are crucial results.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Xxfbus6iCzk", "reviewer": "Reviewer_yiJu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses a problem arising from the generality of source-based conventional batch normalization (CBN) or test-based batch normalization (TBN), with the former biasing the architecture to source distribution and the latter suffering from inaccuracies for small test batch sizes. The paper proposes combining CBN and TBN, and exposing a fraction of different layers in the architecture to TBN and CBN, thereby identifying those layers which need not be as sensitive to the test distribution. The results show promising performance of the proposed method, TTN, on the CIFAR and some DG benchmarks across several batch sizes.\n", "summary_of_the_review": "Overall, I would rate the paper as below the acceptance threshold. While the method appears to show promise in certain settings, I would urge the authors to consider a wider selection of baseline architectures. Also, the results are not convincing enough about its use in a wider array of datasets or real-world scenarios (see the weakness section above).", "strength_and_weaknesses": "Strengths:\n\nThe technique is clear and fairly straightforward to implement.\nMotivation describing pitfalls of CBN and TBN, and the need for better online test-time adaptation under flexible batch sizes are described well\n\nWeaknesses:\n\nIt is unclear how the results would generalize across architectures. What will the results look like for transformers, which are currently ubiquitous in several applications? Will some tokens in a layer be adapted to the test set and the rest to the source? What would this mean in practice?\n\nIn Figure 2, why cannot alpha be learnt or adapted online? If it can be, then these methods should be compared with prompting (freezing certain layers will be analogous to using source-based normalization, while the prompting layers will be adapted to test set). It is unclear why an alpha learnt offline and frozen will continue to yield optimal results across a range of different test-time distributions.\n\nThe significance of results from Table 6 is not clear. These appear to be tested specifically on batch size 2 where TBN is  known to falter, but the results on only a few datasets appear significant. Similarly for the corruption benchmark, with a batch size of 1 or 2 (which is a fairly common real world scenario - data streaming and not evaluated in batches), TTN seems to be beat by one of the baselines. A better understanding and explanation of this is required. Is this because of optimizing the alphas offline?\n\nFinally, what are the restrictions of how far the data distribution that the TTN model is optimized on and the real streaming test time data are? The results appear to indicate that there will be scenarios and assumptions mainly surrounding the dataset that alpha is optimized upon.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "R0rzz1DqVXH", "reviewer": "Reviewer_EgJB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a test-time normalization (TTN) method that combines the normalization statistics of source domain and target domain (during test), by adjusting the importance between them according to the domain-shift sensitivity of each BN layer. The motivation of the proposed method is clear, and the comprehensive experiments conducted in this paper demonstrate the effectiveness of the proposed method for test-time adaptation, particularly in the scenes for small batch size adaptation.  ", "summary_of_the_review": "An empirical successful method for test-time adaptation but with incremental technical novelty and contribution. I am slightly positive to this paper, considering its empirical effectiveness, but not confident to accept it.  ", "strength_and_weaknesses": "\n**Strengths:**\n\n1.The motivation is clear, and the effectiveness of the proposed method makes sense, based on the previous works.\n\n2. This paper provides a comprehensive discussion to the related work, as to me. \n\n3. This paper conducts comprehensive experiments, and the experiments demonstrate the proposed method is empirically successful for test-time adaptation, particularly in the scenes for small batch size adaptation.  \n\n**Weaknesses:** \n\n1.The technical novelty and contributions are incremental. Test-time adaptation using BN is recently widely investigated in previous works, as stated in the related work. Furthermore, the idea of combining the statics of BN from the source domain and targeted domain is not new from previous work. Besides, the method (calculate the domain shift sensitivity) to obtain the prior of $\\alpha$ is from Choi et al. (2022), the optimization (for the interpolating weight $\\alpha$) using cross entropy is also shown in several previous work (even though not specific to the interpolating weight $\\alpha$). I donot well recognize new point for this paper. \n\n2. It should provide the code (including script with hyper-parameters), since this paper conducts comprehensive experiments and the configs of these experiments are complicated. Even though this paper provides details for setup of experiments in the supplementary material, I believe it is not easy to reproduce the results based on these descriptions. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0k4cQc6Thz", "reviewer": "Reviewer_4Hgx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a test-time batch normalization method to tackle domain shifts. The core idea is to learn interpolating vectors to combine running statistics and test-time batch statistics in batch normalization layers. The authors also propose to use a gradient distance score to initialize and regularize the interpolating vectors during training. The proposed method achieves promising results on several image classification and segmentation benchmarks. The authors also conduct extensive ablation studies to validate the design choices.", "summary_of_the_review": "Overall, I think this paper tackles an interesting and important task (test-time adaptation), and the proposed method is well-motivated and effective. I tend to accept this paper. But I suggest the authors add some comparisons and discussion with the missing references I mentioned above.\n\n-----\n\nPost-rebuttal\n\nAfter reading other reviews and the response, I decided to increase my rating.", "strength_and_weaknesses": "**Strength**\n- The proposed method is simple yet effective\n- The proposed method can handle various different test settings and achieve promising performance\n- The authors conduct detailed ablation studies to help the readers to better understand the method\n\n**Weakness**\n1. Missing references\n- [A1][A2] are two recent state-of-the-art test-time adaptation methods, it would be good to add and compare with them\n- The idea of learning interpolating weight is quite similar to [A4]. I know [A4] should be counted as concurrent work and the authors do not have to cite it. But I suggest the authors add and compare with it, to provide a better context to the readers in this area. \n2. More comparisons\n- I think (Schneider et al. 2020) is a strong baseline and should be included.\n- The two state-of-the-art contrastive learning-based methods should also be compared\n- Maybe add more comparisons for the semantic segmentation experiments as well. For example, [A3][A4].\n3. Others.\n- Which TBN method is used in Figure 1? According to my experience, at least (Schneider et al. 2020) should have a reasonable performance (not significantly worse than CBN). Please add more details here.\n- The authors propose a mixed domain evaluation setting, where each batch contains a different domain. However, it is also possible that a single batch contains multiple domains of data. I wonder whether the proposed method can handle this case. [A3][A4] can both handle this and I think the authors can have some study or discussion about this setting.\n- How to select the $\\lambda$ in Eqn. (6) to balance the two loss terms?\n- It is quite interesting to see that the proposed method actually performs worse than non-learning-based normalization methods when the batch size is very small (2 or 1). Can the authors comment on that? Will a dynamic batch size in the post-training stage improve the performance in this case?\n- Why the normalization methods are not shown in Table 2 and Table 3? This looks like normalization methods cannot be used in these settings, but actually they can.\n- While adding the proposed method improves upon the two optimization-based methods in Table 3, the results are even worse than applying the proposed method alone. This may weaken the proposed method and probably the authors can move it to the appendix.\n\n**Reference**\n- [A1] Liu et al. TTT++: When Does Self-Supervised Test-Time Training Fail or Thrive? NeurIPS 2021\n- [A2] Chen et al. Contrastive Test-time Adaptation. CVPR 2022\n- [A3] Khurana et al. SITA: Single Image Test-time Adaptation.\n- [A4] Zou et al. Learning Instance-Specific Adaptation for Cross-Domain Segmentation. ECCV 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster