Test-time Batch Statistics Calibration for Covariate Shift
Authors: fuming you
Jingjing Li
Zhou Zhao
Venue: ICLR 2022 Submitted (2022)

Abstract: Deep neural networks have a clear degradation when applying to the unseen environment due to the covariate shift. Conventional approaches like domain adaptation requires the pre-collected target data for iterative training, which is impractical in real-world applications. In this paper, we propose to adapt the deep models to the novel environment during inference. An previous solution is test time normalization, which substitutes the source statistics in BN layers with the target batch statistics. However, we show that test time normalization may potentially deteriorate the discriminative structures due to the mismatch between target batch statistics and source parameters. To this end, we present a general formulation $\alpha$-BN to calibrate the batch statistics by mixing up the source and target statistics for both alleviating the domain shift and preserving the discriminative structures. Based on $\alpha$-BN, we further present a novel loss function to form a unified test time adaptation framework Core, which performs the pairwise class correlation online optimization. Extensive experiments show that our approaches achieve the state-of-the-art performance on total twelve datasets from three topics, including model robustness to corruptions, domain generalization on image classification and semantic segmentation. Particularly, our $\alpha$-BN improves 28.4\% to 43.9\% on GTA5 $\rightarrow$ Cityscapes without any training, even outperforms the latest source-free domain adaptation method.

Under review as a conference paper at ICLR 2022
TEST-TIME BATCH STATISTICS CALIBRATION FOR COVARIATE SHIFT
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks have clear degradation when applying to the unseen envi- ronment due the covariate shift. Conventional approaches like domain adapta- tion requires the pre-collected target data for iterative training, which is impracti- cal in real-world applications. In this paper, we propose to adapt the deep models to the novel environment during inference. An previous solution is test time nor- malization, which substitutes the source statistics in BN layers with the target batch statistics. However, show that test time normalization may potentially deteriorate the discriminative structures due to the mismatch between target batch statistics and source parameters. To this end, we present general formulation o-BN to calibrate the batch statistics by mixing up the source and target statistics for both alleviating the domain shift and preserving the discriminative structures. Based on a-BN, we further present a novel loss function to form unified test time adaptation framework CORE, which performs the pairwise class correlation online optimization. Extensive experiments show that our approaches achieve the state-of-the-ane performance on total twelve datasets from three topics, including model robustness to corruptions, domain generalization on image classification and semantic segmentation. Particularly, our a-BN improves 28.4% to 43.9% on GTA5 + Cityscapes without any training, even outperforms the latest source-free domain adaptation method.
INTRODUCTION
Deep neural networks (DNNs) achieve impressive success across various applications, but heavily rely on the independent and identical distribution (i.id.) assumption. However, in real-world appli- cations, the model is prone to encounter the novel instances. For examples, an automatic pilot should have robust performance under different weather conditions. Unfortunately, when applying DNNs to novel environment, the performance has a clear degradation due to the covariate shift (Ben-David et al. 2010), i.e., the test data distribution differs from the training distribution.
Domain adaptation (DA) is a promising alternative. which transfers the knowledge learned on la- beled source domain unlabeled target domain, where the data distribution is distinct (Long al.) 2013). However, domain adaptation needs the pre-collected target data, which is not applicable. Un- like DA, Domain generalization (DG) aims at training a general model from multiple source domains and generalizing to the unseen target domain. DG is more challenging since the target domain is to- tally unseen during training. Recently, another practical scenario named test-time adaptation (TTA) is proposed. In TTA, the target data are not pre-collected for iterative training, but used for adapting the source-trained model during inference. We show the comparison between different settings in Table [] To make the comparison clear, we introduce two indicators. Iterative training means the model trained on the unlabeled target data iteratively. Online training means the model parameters updated during inference. Noticing that a recent DG work (Pandey et al. (2021) adapts the model during inference, while most previous DG methods did not. We call it as optimization- free TTA. In this paper, focus the practical scenarios: DG and TTA.
One of the main approaches to adapt during inference is test-time normalization (T-BN) Nado et al. 2020). T-BN re-calculates the target batch statistics to replace the source statistics in BN layers during inference. Motivated by T-BN, Wang et al. (202T) proposed to perform test time adaptation by re-calculating the target batch statistics and updating the affine parameters in BN layers with

---

Under review as a conference paper at ICLR 2022
Table 1:
The comparison between various settings. DG and TTA significantly differ from other settings since
both them get rid the iterative training on target data.
Setting
Source data Ee. M/
Target data
Iterative training
Online training
domain adaptation (DA)
source-free domain adaptation (SFDA)
domain generalization (DG)
X
fully test-time adaptation (TTA)
221
PACS
Office-Home
Source
T-BN
bobd
paiitiig
Clipurt
Cabaar
Cloat
Product
(6) Exomples from Office-Homee and PACS
(b)
Repeesemation 'Cipprr'
(Best viewed in color.) (a) Examples form two DG datasets: PACS and Office-Home (b) Error of the ideal target hypothesis. "Art painting" indicates that the model is trained on the remain- ing source domains: Cartoon, Photo and Sketch, and the target representations are obtained by trained model. (c) Visualization of the target representations by t-SNE (Van der Maaten &
Figure 1:
Hinton 2008). The category cluster T-BN shows larger variance.
entropy minimization. However, this paradigm has critical restrictions. For representation learning, the discriminative representations crucial for recognition task. Substituting the source statistics with target statistics BN layers will inevitably lead to a mismatch with the source-trained model parameters. This mismatch will probably perturb the original discriminative structures. Another limitation is the estimation error on target batch statistics. The source statistics in BN layers are updated in moving average manner during training time, while the target statisties calculated in each batch during test time. The statistics estimated in a batch introduces more errors in reflecting the domain characteristics. preliminary empirical investigation of the mentioned restrictions shown in Fig. and Table 2
Motivated by the hidden restrictions of T-BN, we propose a more general method named o-BN to calibrate the batch statistics during inference. Specifically, we mix up the source and target statistics BN layers to both alleviate the domain shift and preserve the discriminative structures. Equipped with a-BN, common DG models can be further improved without any training. Based on a-BN, we further propose an unified test-time adaptation framework named CORE with an online optimization, which exploits the pairwise Class correlation to facilitate robust and accurate test time adaptation.
To sum up. have following contributions:
1. We investigate two practical yet challenging transfer learning scenarios: domain gener- alization and test time adaptation, which release the requirement of pre-collected target domain data.
2. Motivated by the hidden restrictions of test time normalization, present general formu- lation a-BN for both alleviating domain shifts and preserving discriminative informations It can be seamlessly incorporated into mainstream deep neural networks to improve the generalization on unseen domains without any training.
3. Based Q-BN, we propose a unified framework CORE for test-time adaptation. CORE optimizes the pairwise class correlation in an unsupervised online learning manner.
4. We conduct numerous experiments on total twelve datasets from three topics: robustness to corruptions, DG image classification and DG semantic segmentations. The empirical results show that both G-BN and CORE achieve the state-offther of-the-art (SoTA) performance in their respective communities. The result on GTA5 Cityscapes, for instance, is improved
2

---

Under review as a conference paper at ICLR 2022
from 28.4% 43.9% without any training, which even outperforms the SoTA source-free DA method.
RELATED WORKS
Domain Adaptation and Generalization Domain adaptation (DA) enables transferring the knowl- edge from source domain to target domain by jointly optimizing both labeled source data and unla- beled target data. This paradigm has gained a lot of attention in the last decade and various methods are proposed, which can be roughly divided into three categories: metric learning (Long et al..
2015
Sun & Saenko, 2016 Kang et al. 2019; Li et al., 2020), adversarial training (Ganin et al. 2016
Long et al. 2018 Tsai al.) (0188) and self-training Zou et al. (2019); Liang et al. (2020); Ge et al. (2220). However, training with the large amounts source data is infffiient and impractical in many real-world applications. To address this issue, source-free DA is proposed, which adapts to the tar- get domain with target data and the model pre-trained on source domain. Chidlovskii et al. (2016) the first to investigate source-free DA and proposed a denoising auto-encoder for adaptation. SHOT (Liang et al. 2020) is another representative work, which proposed information maximiza- tion loss and cuustrrng-basee pseudo-labeling, However, despite getting rid of the source domain data, source-free DA also requires the pre-collected target domain data for iterative training, limiting its application scenarios.
Therefore, domain generalization (DG) and fully test-time adaptation (TTA) are proposed (Let us elaborate TTA in the next paragraph). Domain generalization aims generalizing the model trained on (multiple) source domain (s) to the unseen target domain directly. Recently, various DG meth- ods are proposed including domain-invarian representation learning (Zhao et al. 2020 Matsuura & Harada 2020). proxy tasks (Carlucci al. [019 Huang al., 2020), augmentations (Volpi et al. 2018 Zhou et al. (021), ) meta-learning (Li et al. 2018 Balaji et 20118) and so on. However, Gul- rajani& & Lopez- Paz (2020) provided DG benchmark named DomainBed for fair comparison and found a well-implemented empirical risk minimization (ERM) model outperforms most DG meth- ods. Recently, [aandyy et al. 2222T) proposed label- preseerieg target projections during inference time for DG. This work differs from most previous works, which focus on learning from source do- mains, while it performs optimization-free TTA during inference. Our proposed a-BN also belongs to it, but the methods differ.
Fully Test-time Adaptation Fully test-time adaptation is proposed by Wang et (022)) which adapts the model to target domain by online training. TTA can be seen as a compromise between source-free DA and DG. Different from source-fres free DA, TTA does not require the pre-collected target domain data but trains on the target data in an online manner. Also, different from DG, TTA allows optimization during test, which introduces additional test time cost but usually guarantees better performance. Therefore, TTA is a really practical scenario since it gets rid of iterative training and yields better performance compared to generalizing to the new environment directly. [Wang et al. (2021) proposed TENT to achieve TTA by feature modulation. Feature modulation contains two steps: test-time normalization (we will elaborate in the next paragraph.) and affine parameters optimization by entropy minimization, which is widely-used regularization term on DA and semi- supervised leamnngg (Grandvalet et al.) (005). Another similar scenario is test-time training, which optimizes the networks before making prediction during inference (unn et al. (201)). Since this setting is weaker than TTA, we mainly talk about TTA in this paper.
Normalization and Adaptation Batch normalization (BN) is widely-used in DNNs nowadays for stable training and fast converge. BN is originally proposed to alleviate the internal covariate shift during training a very deep neural networks loffe & Szegedy (2015). Recently, Schneider et al. (020)) and Nado et al. (2020) discovered that updating the batch statistics during testing improves robustness to common corruptions. In this paper, we call it as test-time normalization. Similar to their works, Wang et al. (2021) proposed feature modulation, which also updates the batch statistics rather than freezes them. The key insight for these methods is that batch statistics are closely related to the domain characteristics (Li et al.) 2016 Pan et al. 2018). Based this finding, [Joon et al. (021)) and Zhou et al. (2021) proposed to synthesise the novel domain styles to facilitate general- ization. Another kind of work focuses
instance normalization (Ulyanov al. 2016). Similarly, Huang & Belongie (2017) found the instance wise statistics are related to instance characteristics (i.e., image styles). Motivated by this finding, Zhou et al. (202T) proposed to generate novel in-
3

---

Under review as a conference paper at ICLR 2022
stances by mixing instance- feature statistics to enhance cuttof-ddistibuttion generalization. It is worth noticing that the aforementioned DG method can be summarized as an augmentation-bed -based technique. Different from them, we propose post-processing method to calibrate the batch statistics on target domain during test time.
UNDERSTANDING TEST-TIME NORMALIZATION
Test-time normalization (T-BN) re-calculates the batch statistics on target domain during inference. Since the batch statistics are closely related to the domain characteristics (Li al., 2016 Pan et al.) 2018), T-BN adapts the model to target domain ex- plicitly. However, every coin has two sides, and T-BN is not free lunch. During training on the source domain, the model parameters are associ- ated with the source statistics. Therefore, substi- tuting the source statistics with the target ones in- evitably results in mismatch with the model pa- rameters, which leads to the degradation of discrim- inative structures. In short, T-BN alleviates the negative effects caused by domain shift, but perturbs the discriminative structures. We report the averaged accuracy of "Source" and "T-BN" on three DG classification benchmarks in Table We observe that the accuracy of "T-BN" is consistently lower than "Source", revealing that substituting the source statistics by the estimated target batch statistics directly is not effective in generalizing the new environment. To further understand T-BN, we begin with the following two perspectives.
Table 2: Accuracies (%) of "Souree" and "T- BN" on three DG classification benchmarks: VLCS, PACS and Office-
Method
VLCS
PACS
Office-Home
Source T-BN
77.2
85.3
66.5
57.9
83.7
63.9
drop
19.3
1.6,
2.64
Error of ideal target hypothesis. Based domain adaptation theory (Ben-David et al. 2010), the domain shift can be reflected by the error of the ideal target hypothesis based on the target represen- tations learned by source model. "target representation" means the representations are obtained on the target domain data. To obtain the ideal target hypothesis, we train new classifier over the tar- get representations with corresponding labels. Two methods are compared: "Source" and "T-BN". "Source" obtains the target representations by the source model directly, while "T-BN" performs test-time normalization. The error the ideal target hypothesis is shown in Fig. (b). As expected, the error of the ideal target hypothesis in "T-BN" is lower over all tasks. It is worth noticing that the only difference between them that T-BN normalizes the BN layer inputs by the target statistics rather than the source statistics, and others remain consistent (e.g., the same network architecture and the same network parameters). Therefore, we reasonably postulate that T-BN alleviates the domain shift, which results in the lower error of the ideal target hypothesis.
Representation visualization. Discriminative representation learning is essential for recognition task. The discriminative representation satisfies two basic principles: intra-class tightness and inter- class separation. To qualitatively verify that how T-BN affects the learned representations, we vi- sualize the target representations in Fig. [[ (c). The variance of each category cluster in "T-BN" is significantly larger compared to "Source" which indicates the discriminative structures are injured due to the mismatch between target statisties and source model parameters.
4 TEST-TIME BATCH STATISTICS CALIBRATION
Let ( (. be the source statistics in i-th BN layers. After training on the source domain data, the BN statistics are always fixed for stable inference. However, when encountering the data with distinct distribution, the model performance usually has clear degradation due to the covariate } to replace the fixed BN statistics, which alleviates domain shift but perturbs the discriminative structures. In this paper, we present general formulation a-BN to generalize to new domains while preserve the discriminative structures. a-BN considers both source and target statistics.
shift, T-BN re-calculates the target statistics {46²) o
More specifically, a-BN mixes the source and target statistics during test time. During test time, given target input batch St, a-BN re-calculates the target batch statistics (1t
(6) a. before
4

---

Under review as a conference paper at ICLR 2022
forwarding i-th BN layer. Then, o-BN calibrates the batch statisties as:
1 aap!) + - aa)44
(1)
(1) co! + (1 aa)
where is a hyper-parameter to balance the source and target statistics. Similar formulation is also proposed by /Schneider et al. (2020) for alleviating the estimated error caused by small batch size. However, we discover that even with large batch size (e.g., 200), T-BN also yields inferior performance on the large distribution shift (e.g.. Office-Home), In practice, we set a to 0.9 for clas- sification tasks, and 0.7 for segmentation tasks domain generalization benchmarks. Noticing that a-BN is post-processing method, and can be easily incorporated into mainstream neural networks enhance the generalization performance on unseen domains.
Based on Q-BN, we adopt the class correlation optimization (Jin et al. test-time adaptation:
2020 for robust and accurate
C LOORE EEDee j=1jチ}
where P.j is the averaged softmax output, which indicates the probabilities that the samples in the mini-batch belongs to j-th class. p.jp.y' depicts the correlation between j-th class and j'-th class. Minimizing the class correlation leads to a more confident and accurate predictions like other reg- ularization terms (e.g., entropy minimization). However, theoretically show that CORE loss prevents the easy samples dominate the learning procedure in Appendix thus providing a more effective optimization-
test-time adaptation framework.
In short, we propose a unified framework named CORE for test-time adaptation based on a-BN and class correlation. Firstly, we calibrate the batch statistics with proposed a-BN. Then, we only optimize the affine parameters in BN layers with the loss function presented in Eq. (3). Noticing that CORE fully unsupervised, and only updates the parameters once in each test batch. The test data will not be replayed just like the assembly line products.
EXPERIMENTS
5.1 DATASETS AND TASK DESIGNS
We evaluate our proposed method on 12 datasets, including three categories: robustness to corrup- tions, DG on image classification and DG on semantic segmentation.
Topic 1: Robustness to corruptions. We evaluate the robustness on CIFARIO-C, CIFARI00-C and ImageNet-C. For each dataset, 15 types of algorithmically generated corruptions with five levels of severity are included. For CIFAR 10/1 00-C, we evaluate models on the highest severity corruptions. For ImageNet-C, the results are averaged on five levels of severity, i.c., 75 distinct corruptions.
Topic 2: DG on image classification. We evaluate method on four datasets: VLCS, PACS, Office-Home and DomainNet as suggested by DomainBed (Gulrajani & Lapez-Paz 2020).
Topic 3: DG on semantic segmentation. We adopts two synthetic datasets (i.e., GTAS and SYN-
THIA) and three real-world datasets (i.c., Cityscapes, BDD-100K, Mapillary).
More details about these datasets provided in Appendix. A Based on these datasets, conduct numerous tasks with various base models: ResNet (He et al. 2016). WideResNet (Zagoruyko & Komodakis 2016) with AugMix (Hendrycks* et al. 2020 and DeepLabV3 (Chen et al. 2017).
The task design shown Table
5.2 IMPLEMENTATION DETAILS AND BASELINES
To guarantee the reproducibility of our results, we implement the proposed methods on the widely- used benchmarks. We use RobustBench CCrce et al. 2020) for Topic 1, DomainBed (Gulrajani & Lopez-Paz 2020) for Topic and RobustNet (Choi et al. (022I) for Topic Since the proposed a- BN is training-free, illustrate the implementation details of test-time adaptation method CORE.
5

---

Under review as a conference paper at ICLR 2022
Table 3: Task design.
Cuteeryy
Dutaact
Model
Tiak docrigiion
Namber of tasks
CHARID
WiceReslee 25
Clen CIFAR10 Comppted CHEARIO nughest Jevel)
nésustncee
CIFARIOS4C
Clean CIEAR Comppecal CIEAR (lipheel lextl
ImogeNen-
ResNedU
Imagetet- Comppted lmaggeNe Jevelo)
VLCS
ResNedo0
(LCS, VCS, VLS,
C.5). C.S)
DG imge chesificiio
Offez Bone
ReNNa50
ResNect0
CPR, APR, ACR, ACP) LA C.R.R.
(ACS, PCS. PAS, PMC
DemonnNrr
ResNedt
[IPOsS, CONSSCCR UPQRS, CRyKS, CIPRS, CIPSS, CIPQ - (.....O
DG on semmunte scamenattonn
Icc Thak descippiss"
see "Josk descripton"
ReaNo.S0 S0 Dopplaby33
ResNer50 DeepLabVSe
(GTAS) (SYNTHIA, Citysapps 1000-00K. Mapillrry (GIKS-SYNHHAA] - [CHessapees BDD 100K, Mapilary)
Table 5: Test emror values of different comuptiins CIFARIO-C and CIFARI0O-C. The evaluation is imple-
mented on RobustBench (Croce (0000) for fair comparison and easy reproducing. We compare proposed CORE with T-BN (Schneider et al. 2020) and state-of-the thtte-ff-the-rt art method TENT (Wang et al. (021). The best results are highlighted.
Gzess
Imsuie
Bright
Contrast
Scance T-BN
46.7
I6-C
25.1
19.7
2044
TENT
CHARROO
24.8
235
33.0
12.0
31.8
13.7
15.9
16.2
13.7
7.9
12.1
22.0
17.3
24.2
18.6
COREIOun)
CIPARI-
22.5
20.39
29.80
11.0"
29.2"
12.3P
18.2
14.4"
148"
12.4"
2.2
10.6
28.4
15.3
21.4"
16.3
Source
CI-CRI00-C
65.7
60.1
59.1
$2.0
$1.0
33.6
32.3
41.4
45.2
51.4
31.6
55.5
40.3
59.7
42.4
46.3
T-BN
CIFARICO-C
44.3
44.0
47.3
33.1
45.8
$2.8
33.0
38.4
37.9
45.4
29.8
96.5
40.6
36.7
44
39.1
TENT
CIARRIO C
40.1
39.5
42.1
30,1
42.8
31.2
30,2
31.5
36.9
38.7
28.0
33.6
38L1
33.9
40.8
35.0
CORFOORSS)
CIFARICOA
3038
39.34
41.5
29.5
41.7
30.6"
29.8
34.2
34.9
30.6"
27.5"
32.6"
37.15
32.7'
40.1"
35.3
We use SGD optimizer for ImageNet-C, and Adam optimizer (Kingma & Ba 2015) for remain-
ing tasks. We set batch size (BS) as 200, learning rate (LR) as 0.001 for CIFARIO/100, which is consistent with TENT's settings" For ImageNet-C. with other methods. For DG on image classification, we set BS=64, LR=1e-4 for all tasks except Net,in which reduce the LR le-5.
set BS=64, LR=2.5e-4 for fair comparison
To verify the effectiveness of proposed two methods: a-BN and CORE, we compare them with the state-of-the-art methods each research community. For Topie test-time normalization (T- BN) (Nado et al.) (020)) and test-time adaptation by entropy minimization (TENT) (Wang et al. 2021) are noteworthy approaches. For Topic incorporate a-BN with two baselines: Empirical Risk Minimization (ERM) and CORrelation ALignment (CORAL) (Sun & Saenko 2016). For Topic 3, we apply a-BN to the simplest method ERM to enhance the generalization on unseen domains.
5.3 ROBUSTNESS TO CORRUPTIONS
The results of CIFARIO-C and CIFARI00-C are shown in Table 5 Our proposed loss consistently outperforms all existing approaches under TTA set- ting, which demonstrates the effectiveness of our method. To verify the performance on large-scale dataset, we evaluate our method on ImageNet-C, whose results are shown in Table dataset ImageNet-C, we achieve a new state-of-the- art: 42.5% mean error over 75 tasks, which proves the superiority of our method on improving the ro- bustness to corruption by test-time adaptation. worth noticing that we optimize the affine parame- ters in BN layers by only one iteration in an online manner. Therefore, the improvement is significant and the proposed framework CORE shows clear ad- vantages over previous state-of-the-arts like TENT.
Test error values of different cor- ruptions on ImageNet-C. The best results are highlighted. The reported results are aver- aged over total 75 tasks. Detailed results of each task are shown in Appendix B.
Table 4:
the largest
Method
Error (%)
Source
59.5
AugMix (Hendrycks* et al. 2020
51.7
Norm (Sehneider et al. 2020
49.9
ANT (Rusak al. 2020
50.2
ANT+SIN (Geirhos al.
2019
47.4
TENT Wang et al. 2021
44.0
CORE (ours)
42.5*
https://gi wtw.hh/ww...oo/gguuuunnnantttttt

---

Under review as a conference paper at ICLR 2022
Table 6: Accuracies of statc-of-the-art methods four datasets. The evaluation implemented Do-
mainBed (Gulrajani & Lopez-Paz) 2020). The best results on Optimization- Test-Time Adaptation (OF- TTA) and optimization- basod Test- Time Adaptation (TTA) are highlighted by bold and underline, respectively.
Method
Protocols
VLCS
PACS
Office-Home
DomainNet
Mcan
ERM
DG
77.5
85.5
66.5
40.9
67.6
CORAL (Sun & Saenko.
2016
DG
78.8
86,2
68.7
41.5
68.8
Mixup (Xu el al. 2020
DG
77.4
84.6
68.1
39.2
67.3
SagNet
DG
77.8
86.3
68.1
40.3
68.1
MLDG [Li al. 2018
77.2
84.9
66.8
41.2
67.5
RSC (Huang et 2020
77.1
85.2
65.5
38.9
66.7
ERM
OF-TTA
77.2
85.3
66.5
40.9
67.5
ERMi+a-BN (ours)
OF
77.8*
87,9
68.4
41.0
68.8
CORAL#
OF-TTA
78.2
86.1
68.4
68.5
CORAL + (ours)
OF-TA
78.7
87.4"
69.8
41.5
69.4
ERM++ o- BN TENT Wang al. 2021
TTA
78.7
89.1*
67.8*
35.4
67.8
CORAL a-BN TENT Wang al. 2021
TTA
78.9
88.5*
69.9*
38.0*
68.8
ERM++ CORE (ours)
78.4*
89.4*
69.1*
43.8
70.2
CORAL CORE (ours)
TTA
79.2*
88.7*
70.4
43.6*
70.5
5.4 DG ON IMAGE CLASSIFICATION
In this topic, there are two proposed methods: 'ERM+a-BN and 'ERM+CORE' with different evaluation protocols. Firstly, the comparison between 'ERM+a-BN' and other DG methods is fair since o-BN belongs to optimization -fre TTA that only calibrates the batch statisties during infer- ence without any training. Since "ERM+CORE" belongs to optimization-be based TTA, we compare it with previous state-of-the-art work TENT, Noticing that TENT is based on T-BN but T-BN has infe- rior performance on large distribution shift (see Table 2). we equip TENT with the proposed o-BN for better comparison with our CORE. To emphasize the proposed a-BN and CORE are model- agnostic, also CORAL (Sun & Saenko 2016) another baseline. The results are shown in Table
a-BN improves the performance unseen target domain. Domain generalization is really challenging task and many recent proposed methods have few improvements or even degradations compared ERM. Equipped with our proposed a-BN, the performance unseen target domain is consistently improved. For instance, ERM+a-BN reaches 68.8% and CORAL+a-BM reaches 69.4% averaged accuracy over four DG benchmarks.
CORE outperforms TENT with robust performance. Equipped with a-BN, we compare the pro- posed method CORE with TENT, the state-of-the-art algorithm in test-time adaptation. CORE con- sistently outperforms TENT over four benchmarks, which verifies the effectiveness of our method. Specifically, CORE surpass TENT by considerable margins of 2.4% and 1.7% averaged accuracy on ERM and CORAL baselines. Noticing that test-time adaptation only updates the parameters once, thus the improvement is significant. Another interesting finding that CORE surpasses TENT by 8.4% and 5.6% accuracy the most challenging dataset DomainNet.
5.5 DG ON SEMANTIC SEGMENTATION
To evaluate the universality of proposed methods, we conduct DG experiments on semantic seg- mentation with five large-scale datasets: GTAS, SYNTHIA, Cityscapes, BDD-100K, Mapillary. The results are shown Table 7 and the segmentation visualization is presented Fig.
a-BN is embarrassingly simple but achieves state-of-the-art performance. Previous state-of- the-art method ISW proposed instance selective whitening to remove the domain-specific styles, and reaches 40.0% mloU over all tasks. The popular IBN-Net achieves 38, 2% mloU by utilizing both batch normalization and instance normalization. Different from previous works, we propose a proceprocessing method named a- c-BN, which adapts the batch statisties during test time. Equipped with a-BN, the baseline ERM reaches 40.7% mloU over total nine tasks, yielding a new state-of- the-art domain generalization.

---

Under review as a conference paper at ICLR 2022
Table 7: Results semantic segmentation under domain generalization setting. We evaluate method on two popular simulation-to-reat benchmarks. 'I' means results are based on our implementation. The evaluation metrics is the mean intorsection-over-miion (mloU). We compare our method with recent statc-of- the-art methods: SW (Pan al. 2019), IBN-Net (Pan et al. 2018). IterNorm [Hang et al. 2019). ISW (Choi al. 2021> and T-BN Nado al. (2020)
Source
(TTS
GITAS SYNTHIA
Ctysape
Target
SYNTHIA
Citscapes
BODK
Mapulary
Citycapes
BDD-L03K
Mapillary
BDD-100K
Mapillary
MEAN
FRM
20,0
SW
27.6
23.9
27.5
29.7
28.7
48.3
55.8
IBN-Nii
27.9
33.9
32.3
37.8
33.0
35.6
32.2
36.1
35.3
15.6
$7.0
52.8
362
31.8
31.4
BRMI
26.7
25.4
21.3
27.9
26.8
35.2
24.3
31.5
30.7
15.9
52.5
19.2
33.1
LBN
27.2
51.0
32.5
38.1
34.2
66.0
29.3
39.2
47.6
Image
ERM (Source only)
ERM+a-BN
Ground truth
Figure 2: (Best viewed in color.) Qualitative results GTA5 Cityscapes. Equipped with a-BN, the source model yields better and cleaner segmentation map on target domain with little additional test time (e.g., about 16ms each image). Specifically, without any training, a-BN discovers the missing class "person", which is colored red. More qualitative results are shown in Appendix D.5
a-BN even outperforms the source-free DA method without any training. Source-free DA (Liang et al. 2020) becomes a popular topic in DA community recently. For semantic seg- mentation, SFDA (Liu al. 202T) shares similar backbone ours, and reaches 43.2% mloU with the 34. % mloU ERM baseline on GTA5 > Cityscapes. With weaker ERM baseline (28.3% mloU), the proposed o-BN even achieves better performance (43.9% mloU).
ANALYSIS
a-BN introduce little additional inference time. To evaluate the efficiency of proposed a-BN, we calculate the wall-clock time on GTA5 > Cityscapes with the same running environments for five times. The averaged wall-clock times for vanilla inference and a-BN are 72.84s and 80.94s, respectively. With little additional inference time cost (i.e., additional 0.0158s on each image), the proposed o-BN brings 15.5% mloU improvement GTAS Cityscapes.
Cltyring apes Mary
Cippcaps
Seatnce
T-BN
THO0 190
07 02
05 05 02
80 30
01 05
Ar
Gurak
Figure 3: (Best viewed in color.) Analysis of batch size, parameter sensitivity and LogME score.
Test-time batch size. Fig (a) shows the results under different batch size on four DG segmentation tasks. We observe that the performance is robust test-time batch size (BS). For classification task, the observation is similar (Schneider et al. 2020): the performance begins robust when BS > 64.
The syper-parameter is robust to task. There are only one hyper-parameter in proposed G-BN. To evaluate the parameter sensitivity, we illustrate the results on four segmentation tasks with different values of in Fig, B (b). We observe that segmentation tasks, and we also observe that a = 0.9 robust for DG classification tasks. We also present the comparison with other methods in Fig. 3 (c). The mloU in Fig. ( is averaged four
0.7 is great choice for all DG
8

---

Under review as a conference paper at ICLR 2022
tasks (GTA5) - (SYNTHIA, Cityscapes, BDD-100K Mapillary). We observe that a-BN (a outperforms all alternatives. It is worth noticing that a-BN is grainingfrree, and only introduces little additional inference time based on the ERM mdoel, while other methods like ISW has additional loss functions and augmentations.
0.7)
o-BN gains more transferable representations. The LogME score is practical assessment for the transferability of representations (You et al. 2021): Fig. (d) shows the LogME score on tasks (P.C.S) - {A]. [P.A.C] + {S} with the obtained representations of Source, T-BN and the pro- posed a-BN. We observe that the LogME score on a-BN representations is larger than the LogME score on Source and T-BN representations. The finding implies that o-BN gains more transferable
representations.
a-BN reaches statistical significance. To verify the statistical significance of a-BN, we conduct the McNemars Test (Dietterich, 1998) to show the significant improvement brought by the proposed a-BN contrast to the baseline ERM. The p-values for four tasks (the order of tasks is the same as Table B.4) Office-Home are 3,4x10-b, 2.4x 10-13 0.0012 and 0.0016, respectively. As is widely acknowledged, we set the significance threshold as 0.05. Therefore, the significance test indicates our a-BN brings significant improvement to the common baselines like ERM. We also conduct statistical significance test on other benchmarks. The result with *** indicates that it is statistical significant.
7 DISCUSSION
Why CORE (a-BN+CORE loss) outperforms TENT (T-BN+Entropy loss)? Firstly, T-BN leads to a mismatch between target statisties and source-learned parameters, while a-BN balances the co- variate shift alleviation and discriminative structure preservation. More essentially, a-BN provides a better initialization of BN parameters. However, we show that a better initialization is important for test-time adaptation (see Table 8 and 9 Since the supervision signals come from the model predictions, a better initialization leads to a better supervision signals in the online unsupervised learning setting. In Appendix C we provide theoretical insight on why CORE loss outperforms Entropy loss. Compare to Entropy loss, CORE loss has much smaller gradient on the easy sam- ples. preventing easy samples dominate the learning procedure. Extensive empirical results further demonstrate it (see Table 5| and
o-BN v.s. T-BN From Figure 3 (b.c), a-BN has limited improvement compared to T-BN, which is inconsistent with the finding provided in Table 2 In fact, they are not contradictory. Previous work TENT has shown that T-BN performs well on the synthetic-to-real adaptation (Figure (b.c). However, in this paper, we first find that T-BN is not suitable for the nature distribution shift of real-world datasets (Table 2 We think the latter one is more important since distribution shift is common phenomenon in real world. We have also provide the real-to-real adaptation results on semantic segmentation Table 7 We observe that a-BN outperforms T-BN by margin (3.1 mloU).
How to determine the value of a for a new dataset? The value of a plays an essential role in a-BN and CORE. In this paper, perform grid search, and find an empirical rule: 0.9 for classification and a = 0.7 for semantic segmentation (see Figure 3 and 5 Actually, this setting is not an optimal but an acceptable choice for each task, and always outperforms the original T-BN. We think developing the learnable a- a-BN with an automatically adjusted a is an inspiring direction for future work.
8 CONCLUSION
We present a general formulation named a-BN to generalize deep models to the novel environments. By considering both source and target statistics, a-BN alleviates the covariate shift but preserves dis- criminative structures. Based on a-BN, we further propose unified test-time adaptation approach CORE, which provides a robust optimization by investigating the pair-wise correlations. Differ- ent from conventional domain adaptation approaches, a-BN and CORE are online algorithms, which adapts to the target domain during inference. Comprehensive experiments on twelve datasets from three research topics validate the effectiveness and efficiency of our methods.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gener- alization using meta-regularization. In Advances in Neural Information Processing Systems, pp. 998-1008, 2018.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine learning, 1911-2) 2010.
Fabio M Carlucci, Antonio D' D'Innccette, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Do- main generalization by solving jigsaw puzzles. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 2229-2238, 2019.
Liang-Chieh Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05588 2017.
Boris Chidlovskii, Stephane Clinchant, and Gabriela Csurka. Domain adaptation in the absence of source domain data. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 451-460, 2016.
Sungha Choi, Sanghun Jung, Huiwon Yun, Joanne T Kim, Seungryong Kim, and Jaegul Choo. Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 11580-11590 2021.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schicle. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flam- marion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adver- sarial robustness benchmark. arXiv preprint arXiv:2010.09670, 2020.
Thomas G Dietterich. Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, (89):189551923, 1998.
Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. Proceedings of the IEEE International Conference on Computer Vision pp. 1657-1664, 2013.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François training of neural net-
Laviolette, Mario Marchand, and Victor Lempitsky. works. The Journal of Machine Learning Research, 17(1):209622) 2030, 2016.
Yixiao Ge, Dapeng Chen, and Hongsheng Li. Mutual mean-teaching: Pseudo label refinery for unsu- pervised domain adaptation on person re-identification. In International Conference on Learning Representations, 2020. URL httpt:ttts... http::::///peerrriiiieee www.:/eee/e........... www.orrnnnrnrnnnnooooeo
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias im- proves accuracy and robustness. In International Conference on Learning Representations, 2019. URL wtt.httpss//ppeeeeiieeee hwwp:/eee///fooeeeeeee id=Bygh9
Yves Grandvalet, Yoshua Bengio, et al. Semi-supervised learning by entropy minimization. In CAP, pp. 281-296, 2005.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
10

---

Under review as a conference paper at ICLR 2022
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. In International Conference on Learning Representations, 2019. URL
https::/ccpenzet net/
Dan Hendrycks*, Norman Mu Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple method to improve robustmess and uncertainty under data shift. In International Conference on Learning Representations, 2020. URL https: //openrevi ew. fort/forum? wtw.rrmmmmdds11.mmmm g
Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Fsdr: Frequency space domain random- ization for domain generalization. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 6891-6902, 2021.
Lei Huang, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Iterative normalization: Beyond standard- ization towards efficient whitening. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4874-4883, 2019.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance nor- malization. Proceedings of the IEEE International Conference on Computer Vision, pp. 1501- 1510, 2017.
Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 124-140. Springer, 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 4566. PMLR, 2015.
Seogkyu Jeon, Kibeom Hong, Pilhyeon Lee, Jewook Lee, and Hyeran Byun. Feature stylization and domain-aware contrastive learning for domain generalization. arXiv preprint arXiv:2108.08590 2021.
Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum class confusion for versatile domain adaptation. In European Conference on Computer Vision, pp. 464-480. Springer, 2020.
Guoliang Kang, Lu Jiang. Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Pruceedings the and Pattern Recognition, pp. 4893 4902, 2019.
the IEEE Conference on Computer Vision
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta- learning for domain generalization. In Thirry-Second AAAI Conference Artificial Intelligence, 2018.
Jingjing Li. Erpeng Chen, Zhengming Ding, Lei Zhu, Ke Lu, and Heng Tao Shen. Maximum density divergence for domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Yanghao Li. Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normaliza- tion for practical domain adaptation. arXiv preprint 1rXiv:1603.0443 2016.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. arXiv preprint arXiv:2002.08546 2020.

---

Under review as a conference paper at ICLR 2022
Yuang Liu, Wei Zhang, and Jun Wang. Source-free domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 1215-1224, 2021.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105. PMLR, 2015.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pp. 1645-1655, 2018.
Toshihiko Matsuura and Tatsuya Harada. Domain gencralization using mixture of multiple latent
domains. In AAAI, pp. 1179--17766, 2020.
Zachary Nado, Shreyas Padhy, Sculley, Alexander D' D'Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963, 2020.
Hyeonseob Nam, HyunJae Lee, Jongehan Park, Wonjun Yoon, and Donggeun Yoo. Reducing do- main gap via style-agnostic networks. arXiv e-prints, pp. arXiv-1910, 2019.
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEEE international conference on computer vision, pp.
2017.
Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at Enhancing learning and generalization capacities via ibn-net. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 464-479, 2018.
Xingang Pan, Xiaohang Zhan, Jianping Shi, Xiaoou Tang, and Ping Luo. Switchable whitening for deep representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1863-1871, 2019.
Prashant Pandey, Mrigank Raman, Sumanth Varambally, and Prathosh AP. Generalization on unseen domains via inference-time label-preserving target projections. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 19924-12933, 2021.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference Computer Vision, pp. 1406-1415, 2019.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European conference computer vision, pp. 102-118. Springer, 2016.
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3234- 3243, 2016.
Evgenia Rusak, Lukas Schott, Roland S Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. simple way to make neural networks robust against diverse image corruptions. In European Conference on Computer Vision, pp. 53-69. Springer, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):21-252, 2015.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Ad- vances in Neural Information Processing Systems, 33, 2020.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference computer vision, pp. 443-450. Springer, 2016.
12

---

Under review as a conference paper at ICLR 2022
Yu Sun, Xiaolong Wang. Zhuang Liu, John Miller, Alexei A Efros, and Moritz Hardt. Test-time training for out-of-distribution generalization. 2019.
Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker. Learning to adapt structured output space for semantic segmentation. Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7472-7481, 2018.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprini urXi::1607.08022 2016.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using I-sne. Journal of machine learning research, 911)), 2008.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 5018-5021 2017.
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In Advances in neural information processing systems, pp. 5334-5344, 2018.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. International Conference on Learning Repre- sentations, 2021. URL
net/ http://wwweeetooo..ieeeeee.. www.eooooooo....... 3bžLkr3c
Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang- Adversarial domain adaptation with domain mixup. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 6502-6509, 2020.
Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of pre-trained models for transfer learning. International Conference on Machine Learning, pp- 12133-12143. PMLR, 2021.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madha- van, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learn- ing. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp-
2636-2645, 2020.
Sergey Zagoruyko and Nikos Komodakis. arXhn:1605.07146 2016.
Wide residual networks.
arXiv preprint
Shanshan Zhao, Mingming Gong, Tongliang Liu, Huan Fu, and Dacheng Tao. Domain generaliza- tion via entropy regularization. Advances in Neural Information Processing Systems, 33, 2020.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference on Learning Representations, 2021. URL /ttphtttss:://pnerrir
Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5982-5991, 2019.
A DATASETS DESCRIPTION
We evaluate our proposed method on datasets from three communities: robustness to corruptions (CIFAR10/100-C. ImageNet-C), DG image classification (VLCS, PACS, Office-Home, Terra Incognita, DomainNet) and DG on semantic segmentation (GTA5, SYNTHIA, Cityscapes, BDD- 100K, Mapillary). Some examples illustrated in Fig. Here are the detailed descriptions:
CIFAR10/100-C (Krizhevsky et al.) 2009) has 50000 training samples and 10000 test samples with 10/100 classes. The corruptions include gaussian noise, shot noise, impulse noise, defocus blur,
13

---

Under review as a conference paper at ICLR 2022
Geussian Nose
Shot Nose
Impulse Norse
Defocus Buur Prosted Glass
ImageNet-C
Motion Blur
Zoom Blur
Snow
Frost
Fog
GTAS
Cityscapes
Brightness
Contrast
Elastic
Pixelate
JPEG
Mapillary
BDD-100K
real quick guickdrow pointing indogruphh clipr
E
skecch
airplane
clock
axe
bicycle
strawberry
flower
pizza
bracelet
bus
cello
bucket
foot
smorkel
butterfly
cup
Figure 4: (Best viewed in color.) Some examples collected from the datasets: ImageNet-C, Nom GTA5, Cityscapes, BDD-100K and Mapillary. The ImageNet-C figure borrow from (Hendrycks & Dietterich 2019) and the DomainNet figure is borrow from (Peng et al. 2019).
frosted glass blur, motion blur, zoom blur, snow, frost, fog, brightness, contrast, elastic, pixelate and JPEG. (Hendrycks Dietterich 2019)
ImageNet--C (Russakovsky et al. 2015 Hendrycks & Dietterich 2019) has 1.2 million training
samples and 50000 validation samples with 1000 classes. The corruption categories are the same as CIFAR10/100-CC
VLCCS (Fang al. 2013) contains 10,729 examples from four domains: Caltech101 (C), LabelMe (L),, SUN09 (S), VOC2007 (V), with classes.
PACS (Li et al. 2017) contains 9,991 examples from four domains: Art (A), Cartoons (C), Photos (P), Sketches (S), with 7 classes.
Office-Home (Venkateswara et al., 2017) contains 15,588 examples from four domains: Art (A), Clipart (C), Product (P), Real (R), with 65 classes.
DomainNet (Peng et al., 2019) is the largest dataset on DG community, and contains 586,575 ex- amples from six domains: clipart (C), infograph (I), painting (P), quickdraw (Q), real (R), sketch (S), with 345 classes.
GTAS Richter et al. 2016) is large-scale synthetic dataset, which contains 24,966 high-resolution images collected from the game, Grand Theft Auto (GTA5), and the corresponding ground-truth segmentation maps generated by computer graphics. It shares 19 classes with Cityscapes.
SYNTHIA (Ros et al. 2016) is also synthetic dataset rendered from a virtual city and comes with pixel-level segmentation annotations. We adopt the subset of it, which is called WYNTHIA-RAND CITYSCAPES. This subset contains 9400 images and shares 16 common classes with Cityscapes dataset.
Cityscapes (Cordts et al. 2016) is the widely- used real-world dataset, which contains 3,450 high- resolution (i.e., 2048 1024) urban driving scene images.
14

---

Under review as a conference paper at ICLR 2022
BDD-100K (Yu et al. 2020) is recent real-world dataset containing 8000 urban images (7000 images for training and 1000 images for validation) with resolution x280x720
Mapillary (Nuuhold et al. 2017) is another recent real-world dataset containing 25,000 street-view images with minimum resolution 1920x 1080.
15

---

Under review as a conference paper at ICLR 2022
B
FULL RESULTS
We implement our methods with Pytorch on a single NVIDIA RTX 3090 GPU. All results this section are obtained our implementations.
B. IMAGENET-C
All results are statistical significant compared to baseline.
Corruptions
Level
Level 2
Level
Level 4
Level 5
Mean
Guass.
32.33
37.13
45.00
55.15
69.06
47.73
Shot
32.83
38.09
45.08
57.57
67.19
48.15
Impulse
36.05
41.44
45.97
56.16
68.06
49.53
Defocus
34.13
39.33
50.59
60.67
70.60
51.06
Glass
33.26
39.22
52.83
58.89
70.23
50.89
Motion
30.32
34.06
41.03
49.81
57.08
42.46
Zoom
34.83
39.12
41.93
45.66
49.74
42.26
Snow
34.71
43.64
43.36
50.06
51.38
44.36
Frost
34.23
43.70
51.28
52.38
57.61
47.84
Fog
30.23
31.63
33.88
36.05
42.12
34.78
Bright
26.48
27.44
28.73
30.60
33.01
29.25
Contrast
28.73
30.42
33.35
43.01
66.79
40.46
Elastic
30.88
44.32
31.03
34.28
44.17
36.94
Pixel
28.82
29.70
32.93
37.88
41.14
34.09
JPEG
30.37
32.81
34.49
39.68
47.22
36.91
Mean
31.88
36.80
40.77
47.19
55.69
42.47
B.2 VLCS
Algorithm
C
Avg.
ERM
97.5
63.4
73.8
74.0
77.2
ERM+a-BN
96.3*
67.7*
73.5*
73.5
77.8
ERM+TENT
97.3*
69.7*
75.1*
72.8
78.7
ERM+CORE
97.0*
67.7*
76.6*
72.4
78.4
CORAL
97.9
66.1
73.2
75.4
78.2
CORAL+a-Bi
97.0*
69.3*
74.4*
73.9
78.7
CONAL+TENT
97.4*
69.1*
74.5*
74.6*
78,9
CORAL+CORE
97.3*
69.1*
75.7*
74.7*
79.2
B.3 PACS
Algorithm
A
C
P
S
Avg.
ERM
84.7
80.2
96.9
79.2
85.3
ERM+a-BN
88.2*
83.5*
97.5*
82.4*
87.9
ERM+TENT
90.4*
83.5"
97.7*
84.8
89.1
ERM+CORE
90,4*
83.8*
97.7*
85.8
89.4
CORAL
88.2
80.0
97.3
78.8
86.
CORAL+a-BN
89.0*
82.5*
97.7*
80.2*
87.4
CORAL+TENT
90.4*
82.9*
97.9*
82.9*
88.5
CORAL+CORE
90.5*
82.8*
97.9*
83.6*
88.7
16

---

Under review as a conference paper at ICLR 2022
B.4 SFICE-HOME
Algorithm
A
C
P
R
Avg.
ERM
61.2
52.7
75.8
76.3
66.5
ERM+a-BN
63.2*
56.1
76.9*
77.3*
68,4
ERM+TENT
62.6*
54.4*
77.2*
77.1
67.8
ERM+CORE
63.6*
57.9"
77.3*
77.6
69.1
CORAL
64.3
54.7
76.6
78.0
68.4
CORAL+a-BN
64.6*
58.4
77.2*
78.9
69.8
CORAL+TENT
64.7*
58.5*
77.8*
78.7
69.9
CORAL+CORE
65,3*
60.1
77.5*
78.7
70.4
B.5 DOMAINNET
Algorithm
cli
info
paint
quick
real
sketch
Avg.
ERM
57.9
19.1
46.7
12.4
59.6
49.8
40.9
ERM+a-BN
57.9
19.1
46.7
12.4
60.1
49.7
41.0
ERM+TENT
59.1*
11.2*
30.5
1.8*
60.2
49.4*
35.4
ERM+CORE
59.8*
20.7*
50.1*
17.9*
61.5*
53.0*
43.8
CORAL
59.2
19.8
47.3
13.1
59.0
50.2
41.4
CORAL+a-BN
59.2
19.8
47.3
13.1
59.1
50.2
41.5
CORAL+TENT
59.0*
11.7*
44.1*
3.2*
58.5
51.2*
38.0
CORAL+CORE
60.8*
21.3*
48.1*
17.1*
61.5
53.0*
43.6
B.6 DG ON SEMANTIC SEGMENTATION
read
build.
wall
pole
3ky
mnin
bake
nalok:
Metted
SYNTHIA
ERM
45.2
38.4
75.5
45
21.6
13.4
9.3
59.4
$8.8
56.0
51.2
12.8
0.0
7.6
26.7
9-RN
51.2
29.5
50.7
7.6
10.1
2888
15.4
11.2
61.0
$9.5
52.4
7.9
51.5
0.0
25.5
0.0
148
7.3
28.7
+2.0
Cityscapes
ERM
39.7
23.8
$2.9
16.0
17.5
23.8
30.7
14.5
$1.1
27.2
39,8
58.6
6.4
57.2
18.5
14.0
1.0
7.3
9.3
28.4
$7.0
38.7
$3.3
30.3
27.4
35.1
36.4
24.6
$2.8
3000
77.4
65.6
859
30,8
27.7
5.2
16.8
26.3
43,9
+155
GTAS
BDD-LOOK
ERM
43.7
21.7
12.8
3.5
21.0
27.1
29.2
17.3
58.4
21.0
31.0
42.6
4.3
66.9
12.9
4.8
0.0
14.1
10.2
24.3
76.4
76.9
33.4
GTA5
Mapillar
ERM
45,3
24.4
32.7
6.5
17.0
28.0
35.4
8.1
66.0
24.6
40,4
$3.0
4.8
72.4
23.8
5.1
10.1
12.5
20.2
27.9
GTAS SYNTHIA Cryscapps
FRM
74.5
36.7
66.5
11.5
31.4
35.8
21.4
84.7
10.
73.2
66.2
84.6
15.7
25.7
0.0
12.0
21.6
36.2
BN
83.4
$4.0
27.8
11.4
41.2
41.9
32.1
85.7
27.7
$7.3
65.0
19.2
87.0
22.9
31.1
0.1
22.4
34.2
44.8
+12.6
GTAS SYNTHIA
BDD 100K
ERM
40.7
33.3
19
28.8
38.3
19.4
63.6
44.6
$1.8
61.4
4.0
0.0
11.6
24.3
9-BN
74,9
30.4
60.3
5.8
18.4
3688
40.4
32.4
71.3
21.6
77.8
39.9
13,0
76.5
7.6
14.7
0.0
16.2
24.1
35,0
+10.7
GTA5 + SYNTHIA Mapillary
29.8
31.5
IN
74,4
39.1
52.1
17.1
15.6
40.4
46.6
44.4
79.2
42.4
72.5
48.3
14.1
77.5
25.3
17.5
1.0
24.7
33.6
40.3
+8.5
17

---

Under review as a conference paper at ICLR 2022
C
THEORETICAL INSIGHT ON CORE LOSS
In this paper, adopt the CORE loss with class correlation optimization rather than the Entropy loss suggested by TENT (Wang et al.] 202T). From Table 4 5 and we observe that CORE loss outperforms Entropy loss a wide range of benchmarks. To better understand CORE loss, provide a theoretical insight on why CORE loss outperforms Entropy loss. Firstly, the CORE loss Eq. (3) equal to:
C LOORE 1
111--ppp P,Ps-
(4)
where Py denotes the probability class j. The entropy minimization loss is defined as:
C
C Entropy Σ log(p,).
(5)
Consider binary classification problem, we can obtain the values and the gradients with respect to Entropy loss and CORE loss:
Eettoop = -plogp p) p)g( log(1 p);
dL Entropy log(1 p) -logp.
dp
LCORE = - p² - p)²
(8)
dLCORE = 4p.
(9)
dp
From Eq. (7) and Eq. (9) the absolute value of Entropy loss gradient on the easy sample (p > 0.9 o < 0.1) much larger than the absolute value of CORE loss gradient. In other words, Entropy loss pays too much attention to the easy samples.
D MORE EXPERIMENTAL RESULT AND ANALYSIS
D.1 ABLATION STUDY
CORE contains two components: Q-BN and Class Correlation Optimization (COO), while TENT also contains two components: T-BN and Entropy Minimization (EM). In Table 6 compare BN + COO with a-BN + EMM, and show that COO significantly outperforms EM. this section, further evaluate remaining
variants: T-BN COO and T-BN EM (Original TENT). From Table 8, we observe that T-BN works badly on the nature distribution shift of real-world datasets. Equipped with T-BN, COO achieves similar performance with EM, and both of them are worse than the ERM baseline. This ablation study indicates that a-BN is really important for adapting the model novel target domain.
Table 8: Results of four optimization-based TTA methods VLCS, PACS and Office-Home.
Dataset
PACS
Office-Home
Algorithm
C
V
Avg.
A
C
Avg.
C
P
Avg-
ERM
97.5
63.4
73.8
74.0
77.2
84.7
80.2
96.9
79.2
85.3
61.2
$2.7
75.8
76.3
66.5
T-EN
66.9
49.3
53.2
62.1
57.9
87.4
81.3
95.6
74.6
84.7
61.2
50.4
72.0
76.1
64,9
T-BN
68.7
50.9
63.6
58.9
87.8
81.5
95.6
75.6
85.1
51.7
73.0
65.7
T-BN COO
68.5
50.3
54.6
62.4
59.0
87.8
81.8
95.6
75.2
85.1
61.8
51.8
73.0
76.6
65.8
CORE (ours)
97.0
67.7
76.6
72.4
78.4
90.4
83.8
97.9
85.8
89.4
63.6
$7.9
77.3
77.6
69.1
18

---

Under review as a conference paper at ICLR 2022
D.2
RESULT ON GTAS CITYSCAPES WITH ANOTHER BACKBONE
DDDepaab is the popular backbone adopted by many domain adaptation methods. To better evaluate our a-BN, we report the performance with this backbone the most popular transfer task: GTA5 + Cityscapes. The mloU of ERM baseline is 35.7%. a-BN achieves 44.7% mloU, which is also competitive state-of-ther -frt DG method FSDR (Huang 44.8% mloU.
2021 with
Table 9: Results of four optimization-based TTA methods GTA5 - Cityscapes.
100000
ERM
23.8
52.9
16.0
17.5
23.8
30.7
14.5
$1.1
27.2
39.8
58.6
6.4
57.2
185
16.0
1.0
7.3
9.3
28.4
78N
$5.4
41.9
$1.5
26.2
23.0
13.3
35.0
20.7
71.9
20.7
75.6
62.2
26.3
53.7
26.4
25.8
1.4
15.0
20.6
41.0
+12.6
7-8N-BM (TENT)
43.9
80.6
24.7
32.8
35.1
12.5
94.8
60.4
23.6
82.6
268
12.4
16.0
+12.0
7-BN-CO0
212
25.
c-BN
38.7
83.3
30.3
27.4
35.1
36.8
26.6
12.8
30.0
77.6
65.6
55.9
308
27.7
5.2
16.8
26,3
43.9
+15.5
6.BN-BM
88.5
40.7
9288
29.4
21.9
33.9
29.7
19.7
83.
302
79.0
65.5
28.1
55.6
300
27.3
1.7
18.0
22.8
43.1
+14.7
BN-CCO (CORE)
58.8
43.4
835
28.6
27.0
40.5
424
33.1
92.7
27.6
77.0
65.6
55.6
255
28.0
2.7
17.7
26.5
45.0
+166
D.3
RESULT OF OPTIMIZATION-BASEI TTA METHODS ON GTAS = CITYSCAPES
We experiment the optimization- based TTA methods TENT, CORE and their variants on the se- mantic segmentation task GTA5 Cityscapes. We set batch size and learning rate le-5 with Adam optimization. The results are shown in T We observe that COO outperforms EM with both T-BN and c-BN. c-BN+COO achieves the best performance, demonstrating effectiveness of each component of CORE. However, the single a-BN has already achieves 43.9% mloU, COO appears have limited improvement, and EM even leads negative transfer. Recent advances in domain adaptive semantic segmentation also revealed that the universal unsupervised loss is not suit- able for semantic segmentation, since this task has many intrinsic challenges like dlas-imbaalance, boundary confusion and so on. Inspired by that self-training has become the dominant method in domain adaptive semantic segmentation, we think a well-designed self-training loss equipped with Q-BNin is hopeful direction for test-time adaptive semantic segmentation.
D.4 PARAMETER SENSITIVITY OF SYPPER-PARAMTIEE
The value of byper-parameter a plays an essential role in a-BN and CORE. In this paper, we de- termine the value of through grid search: from to with an interval of 0.1. The result of gird search on semantic segmentation is illustrated in Figure. 3 For image classification, we find the performance usually drops when a is decreasing from 0.9 to O. Actually, a = 0.9 is not the optimal choice for each task, but acceptable choice. We further provide the result on image classifica- tion Figure. 5 We think developing the learnable a-BN with an automatically adjusted a is an inspiring direction for future work.
100
75
85
95
70
85
60
50
35
45
1
0.5
0.5
09
0.5
09
0.5
Figure 5: (Best viewed in color.) The results of grid search on classification datasets VLCCS, PACS and Office-Home. The horizontal coordinate represents the value of a and the vertical coordinate represents the classification accuracy.
19

---

Under review as a conference paper at ICLR 2022
D.5 QUALITATIVE RESULTS ON GTA5 CITYSCAPES
GT
ERM
a-BN
GT
ERM
C-BN
GT
ERM
C-BN
Figure 6: (Best viewed in color.) Qualitative results on GTA5 - Cityscapes. "GT" means ground truth. Equipped with our o-BN, the segmentation performance new environment is significantly improved with little additional test time cost (e.g., about 16ms on each image).
20


Reviews:
[{"review_id": "rNzz79DiULV", "reviewer": "Reviewer_8DqH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The authors conduct a lot of experiments to validate that simply calibrating the batch statistics at test-time can improve the model performance in different settings, such as robustness to corruptions and domain generalization.\n2. The analysis in Figure 1 is inspiring and interesting.\n\nWeaknesses\n1. Lack of technical novelty.\n- The idea of mixing source and target statistics has been proposed by Schneider et al. (2020). I don’t see how the so-called alpha-BN differs from it.\n- It seems that the class correlation minimization loss is directly taken from Jin et al. (2020). I think the authors should not claim to use it as a contribution.\n2. Method\n- The logic flow from Section 4 to Section 5 is a bit weird to me. We can know replacing source statistics with target statistics can indeed reduce covariate shift if we use that representation to train a classifier (Figure 1(b)), but simply replacing the statistics will hurt the performance (Table 2). Do these two results intuitively conclude that we should mix the statistics from both source and target domains? I don’t really think so. Also, the selection of the alpha values seems so ad-hoc, without any theoretical analysis. What’s more, the alpha value of the semantic segmentation task is kind of overfitting on the test set (Figure 3). Isn’t this cheating?\n- While in the classification problem we know that directly replacing batch norm statistics with target statistics is bad (Table 2), it actually works quite well in semantic segmentation (Figure 3(b)). I wonder why. I think we need more analysis to understand this phenomenon.\n- Can the class correlation minimization loss be applied to the semantic segmentation task as well?\n3. Some claims are wrong\n- In Table 1, not all the domain generalization methods support online adjustment (actually, only one recent method supports this). I would suggest adding another term called test-time batch statistics update (or something similar) for better clarity.\n- In Section 4, the authors mention test-time normalization. But in the related work section, the authors actually use the same term to refer to both Schneider et al. (2020) and Nado et al. (2020). However, these two works apply test-time normalization differently. The former one does a very similar thing as what this paper does, while the latter one replaces batch norm statistics with target batch statistics. It is confusing to use a single term to refer to two different methods.\n4. Some details are missing\n- In Section 4 “Error of ideal target hypothesis”, the author claims that “we train a new classifier over the target representations with corresponding labels”. Please define “target representations” here.\n- In Section 7 “Discovering a better pre-trained model”, how to use alpha-BN to get a better pre-trained model for fine-tuning remains unclear to me.\n5. Other comments\n- The ERM baseline on semantic segmentation is too bad. Currently, GTA5 -> Cityscapes only gives 29% mIoU by using a ResNet-50 DeepLabv3. However, I know that we can easily get a 35~37% mIoU by using a ResNet-50 DeepLabv2, which is actually a weaker model. I think there must be something wrong with either the model setting or training schedule. Showing great improvement over a bad baseline is not that meaningful.\n\n", "summary_of_the_paper": "This paper presents a method to calibrate batch normalization statistics at test time to improve a model’s cross-domain generalization ability under covariate shifts. The authors validate the effectiveness of the method on several datasets and tasks. The authors also conduct some interesting analyses to help better understand the method.", "summary_of_the_review": "- The main concern is the lack of technical novelty. I would say both of the two main components in this method have been proposed by others previously. In this case, this paper over-claims its contributions. It will be better to reshape the paper as an empirical finding paper instead of a method-proposing paper. \n- Also, semantic segmentation is so different from image classification, so that I am not sure if the conclusions we get from image classification can safely transfer to semantic segmentation. For example, the mismatch between Table 2 and Figure 3(b), as I mentioned in the weakness section as well. And it is also quite weird that another important component, class correlation minimization, is only applied on classification but not segmentation. So I would suggest the authors either discard the semantic segmentation and fully focus on image classification, or conduct additional experiments and analysis on semantic segmentation to have a deeper understanding.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mS90fi_6jFt", "reviewer": "Reviewer_H2Ln", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Proposed method can be easily applied on top of any base model for performing test-time adaptation\n- Strong empirical results across a range of datasets and tasks, especially image segementation, outperforming recent methods\n- Some insight into the failures of T-BN is described \n\nWeaknesses:\n- It is unclear how the parameter $\\alpha$ should be set for new datasets. The chosen values happen to achieve the best performance according to Figure 3 (for segmentation), but how was this chosen and how should these be chosen in practice for a new dataset when the test performance is unknown? How $\\alpha=0.9$ was selected for classification tasks was also not described in the text. \n- A major contribution in the paper is the $\\alpha$-BN method of combining source and target batch statistics, which the paper argues improves upon T-BN. However, the precise effect of this procedure versus the orthogonal class correlation optimization (CCO) was not examined in the experiments. In particular, each of the $\\alpha$-BN and CCO components can be incorporated/compared with the previous TENT method that uses T-BN and entropy. An ablation study swapping out each respective component will show the significance of each part (e.g. T-BN + CCO, $\\alpha$-BN + Entropy).\n- There appear to be some inconsistencies with the experimental evaluation as detailed below:\n  * For segmentation experiments, it appears not all of the methods share the same backbone model, which has a large impact on performance, e.g. SW (Pan et al. 2019) reports using \"DeepLab-v2 [2] model with VGG16 [25] backbone\", but this paper uses DeepLabV3 with ResNet-50 as stated on page 8. Moreover, the result for SW in the GTA5 -> Cityscapes setting is 29.9 in Table 7, which differs from the result reported in the paper (Table 5; AdaptSetNet-SW - mIOU 35.7). These differences should be clarified during the discussion period. It should also be clarified whether the same backbone is used for comparisons in all other experiments as well.\n  * TENT is included in other comparisons but not in the segmentation results - is there some particular reason why? The original TENT was also not included in Table 6 - how does it perform?\n- Comparison to recent DG for segmentation baseline is missing - FSDR: Frequency Space Domain Randomization for Domain Generalization, CVPR 2021.\n- Presentation wise, some parts of the text should either be supported by evidence or toned-down in terms of the claims, e.g. \"guaranteed discriminative representations\" (this suggests there will be a proof), \"suppresses the confident false predictions ... providing a more robust optimization\" (some empirical evidence of this should be provided). Also, Section 4 seems a bit tangential to the rest of the paper as it does not directly motivate $\\alpha$-BN. Does the proposed $\\alpha$-BN method perform better than T-BN in evaluations provided in Section 4? This will provide better motivation for the $\\alpha$-BN method and improve the flow of the paper.\n\nOther comments/suggestions:\n- The justification for comparing the proposed method with other DG methods is that Pandey et al 2021 does some optimization at test time. However, they do not modify the model parameters unlike the proposed method and other test-time adaptation approaches. Moreover, the method of Pandey et al 2021 was not included in the comparisons.\n- It would be useful to provide some insight into why the proposed method does well in some cases compared to others.\n- Statistical tests for significance should probably be done to compare all methods rather than just the baseline, and on all tasks.", "summary_of_the_paper": "This paper presents a domain adaptation algorithm for the recently introduced test-time adaptation setting by adapting batch-norm parameters. The algorithm has two key components: first, it adapts batch-norm statistics using a linear combination of source and estimated target domain statistics; second, it uses a class correlation optimzation loss to optimize batch-norm parameters on the test batches. These design choices differ from previously described methods. The resulting algorithm is evaluated on a range of datasets and tasks including common corruptions and the DomainBed benchmark for image classification, as well as a benchmark for image segmentation.\n\n\n", "summary_of_the_review": "Overall this paper proposes a simple and effective method for test-time adaptation that seems to outperform the previously described TENT method. However, key questions about how the $\\alpha$ parameter should be set in practice and issues with the experimental evaluations should be resolved before the paper is ready for publication. A more thorough examination of the effect of $\\alpha$-BN and CCO components will also provide more confidence in the contribution. \n\n**Post response update:** I have read the other reviews and responses from the authors. The additional ablation studies address my concerns about the effect of the individual components, but as pointed out by the other reviewers as well, technical novelty is somewhat limited as both components have existed in the literature although they are not applied to this particular setting. Also, it is unclear how the $\\alpha$ parameter can be set in practice. Thus, while this work presents an interesting empirical study, I think it is borderline and I lean towards rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "imm6Ud4bQNJ", "reviewer": "Reviewer_A9FS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n\nThe paper uses the same setting of the test-time BN methods to perform test time adaptation on unlabeled target mini-batch data. The method has been well-motivated by pointing out the limitations in SOTA methods, i.e., BN adaptation alleviates the domain statistics shift, but perturbs the discriminative structures. The paper provides comprehensive experiment results.    \n\nweakness:\n - They present the $\\alpha$-blending formulation to mix up the source and target statics instead of directly using the target statistics. They empirically set the weight $\\alpha$ =0.9 for image classification and 0.7 for image segmentation tasks. There is no discussion on how to select $\\alpha$. \n\n - The paper proposes to update beta and gamma on the mini-batch with the Core loss. The main concern is that the proposed Core loss in EQ 5,  $L_{core}=  1 - \\sum_j p_j p_j$,  is actually very similar to the entropy loss $L_{ent} = - \\sum_j p_j log(p_j)$. The experimental results in the papers show the performance is better. However, there is no theoretical proof or explanation that the Core loss can necessarily get better results than the entropy loss. And, it has the same problem as the entropy loss: the proposed loss cannot prevent trivial collapsed solutions where all unlabeled samples are assigned the same one-hot encoding (which is the actual minima).", "summary_of_the_paper": "The paper proposes a test-time batch normalization method for domain adaptation. They present a formulation $\\alpha$-BN to calibrate the batch statistics by mixing up the source and target statistics with a fixed hyper-parameter of alpha. And, they propose to use the Core loss [ Jin et al. (2020)] to optimize the affine parameters of beta and gamma in BN layers instead of the entropy minimization used in the Tent method. The paper provides comprehensive experiment results.    ", "summary_of_the_review": "The proposed method can be considered as an extension of Tent. Essentially, the optimal BN parameters ( $\\mu$, $\\sigma$, $\\beta$, and $\\gamma$) can be obtained by optimizing only the BN affine parameters ( $\\beta$, and $\\gamma$). The proposed alpha-BN actually provides a better initialization of the BN parameters for DA, which is why it can obtain better performance than T-BN. But, there is no proof that the proposed $L_{core}$ can necessarily achieve better results than Tent. I consider the proposed method is incremental. The authors should provide more explanation & discussion of the $L_{core}$ in the rebuttal. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VZx2WbuUO4n", "reviewer": "Reviewer_cBnx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n1. Both proposed modifications are technically sounds. \n\n2. Empirical evaluation on many experimental settings show reliable (yet not revolutionary) performance improvement by the proposed methods. \n\n3. This paper is well written and easy to follow. \n\n\n**Weaknesses**\n\n1. The proposed method is technically sound but not a revolutionary idea. Regarding CORE, difference from (Jin et al., 2020) is not clear from the manuscript. \n\n2. Table 1 is confusing, since DG usually assumes the existence of source data, and does not adapt online usually. Similarly, protocol DG in Table 6 is a little all methods listed in the first block of Table 6 does not use any test-time adaptation. I think it is better to call it optimization-free test-time adaptation to clarify the difference with optimization-based TTA like TENT. \n\n3. From my personal experience, I am a little bit  surprised that simple T-BN significantly worsened the performance in Table 2. In my past implementation, I also observed slight performance degradation on VLCS (say 3%) yet slight performance improvement on PACS (say 1.5%) by T-BN. \nWhat is the batch size in the experiment? Is there any class imbalance among test bach in your evaluation? Any other detail that could change the results?\n\n4. Table 6 shows that using Tent in DomainNet worsens the performance yet CORE stable increases the performance. They claim that this is because the domain gap in DomainNet is extremely large, which might be a true yet too casual statement not suitable for the academic paper. It should be justified qualitatively or quantitatively why Tent fails to handle the huge distribution gap, and DomainNet has a huge distribution gap. \nBesides I am suspicious that the performance gain is caused by the hyperparameter selection. It should be better to include the hyperparameter sensitivity of Tent and CORE to fully show the CORE is indeed superior one. \n\n5. Table 7 lacks comparison with T-BN, therefore it is not clear whether the performance gain caused by the modification or the nature of test-time normalization. Besides, If I correctly understand it, CORE can not be used in semantic segmentation (as with the Tent can not), which should be elaborated in the manuscripts.  \n\nA minor comment: In the last sentence in 6.4, I think Tent should be CORE. \n", "summary_of_the_paper": "This paper presents several modifications over test-time normalization and test-time adaptation, which swap the statistics of the BN layer during test-time to handle the distributional difference. Specifically, they propose alpha-BN to balance the source statistics and target statistics, and CORE to further refine the transformation parameters of BN, which is similar to prior test-time adaptation methods. \n", "summary_of_the_review": "Overall I think this is well-written and an acceptable paper with solid empirical results. I am happy to increase my score if the response clarifies my concerns or I misunderstood some points. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper builds on ideas in test-time adaptation and test-time normalization to improve performance under covariate shift. Concretely, the paper proposes (i) alpha-BN, a method to calibrate batch statistics by mixing source and target statistics and (ii) test-time adaptation using the CORE loss (which was proposed by Jin et al., 2020). The authors compare the the proposed approach to existing approaches on multiple benchmarks. 

The reviewers found the idea interesting and appreciated the additional ablations. The main concerns were around novelty (as the idea is closely related to prior work in test-time adaptation and normalization) and hyperparameter selection (e.g. how to choose alpha in practice).  Overall, the reviewers and I felt that the current version falls slightly below the acceptance threshold. I encourage the authors to revise and resubmit to another venue. 

Minor comment about Appendix C (this didn't affect the score, just a suggestion for future revisions): 
I think it might be interesting to include other alternatives to cross-entropy that downweight easy examples, cf. focal loss https://arxiv.org/abs/1708.02002 and https://arxiv.org/abs/2002.09437. I'm curious to see if CORE and focal loss consistently outperform cross entropy.
Decision: Reject