Adaptive Generalization for Semantic Segmentation
Authors: Sherwin Bahmani
Oliver Hahn
Eduard Sebastian Zamfir
Nikita Araslanov
Stefan Roth
Venue: ICLR 2022 Submitted (2022)
Keywords: domain generalization
semantic segmentation
test-time training

Abstract: Out-of-distribution robustness remains a salient weakness of current state-of-the-art models for semantic segmentation. Until recently, research on generalization followed a restrictive assumption that the model parameters remain fixed after the training process. In this work, we empirically study an adaptive inference strategy for semantic segmentation that adjusts the model to the test sample before producing the final prediction. We achieve this with two complementary techniques. Using Instance-adaptive Batch Normalization (IaBN), we modify normalization layers by combining the feature statistics acquired at training time with those of the test sample. We next introduce a test-time training (TTT) approach for semantic segmentation, Seg-TTT, which adapts the model parameters to the test sample using a self-supervised loss. Relying on a more rigorous evaluation protocol compared to previous work on generalization in semantic segmentation, our study shows that these techniques consistently and significantly outperform the baseline and attain a new state of the art, substantially improving in accuracy over previous generalization methods.

Under review as a conference paper at ICLR 2022
ADAPTIVE GENERALIZATION FOR SEMANTIC SEGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Out-of-distribution robustness remains a salient weakness of current state-of-the- art models for semantic segmentation. Until recently, research on generalization followed restrictive assumption that the model parameters remain fixed after the training process. In this work, we empirically study an adaptive inference strategy for semantic segmentation that adjusts the model to the test sample before producing the final prediction. We achieve this with two complementary techniques. Using restance-adaptive Batch Normalization (laBN), we modify normalization layers by combining the feature statistics acquired at training time with those the test sample. We next introduce a test-time training (TTT) approach for semantic segmentation, Seg-TTT, which adapts the model parameters the test sample using a self self- supervuee loss. Relying on a more rigorous evaluation protocol compared to previous work on generalization in semantic segmentation, our study shows that these techniques consistently and significantly outperform the baseline and attain a new state of the art, substantially improving accuracy over previous generalization methods.
INTRODUCTION
The current state of the art for semantic segmentation (Long et al., 2015 Chen et al. 2018b) lacks direly in out-of-distribution robustness, i. e. when the training and testing distributions different. Numerous studies have investigated this issue, with primary focus on image classification (Arjovsky ct al. 2019] Bickel et al. 2009 Li al. 2017a; Torralba and Efros) [011] Volpi al. 2018). However, the conclusion from a recent study of domain generalization methods (Gulrajani and Lopez-Paz
2020), spanning more than three years of research, disillusioning: Empirical Risk Minimization (ERM), which is based on the i. i. d. assumption of the training and testing distributions, is still highly competitive. This is in stark contrast the evident advances in the area of domain adaptation, both for image classification (Ben-David et al., 2010 Ganin et al. [0116; Long et al. 20166] Xie et al. (018) and semantic segmentation (Araslanov and Roth 2021 Yang and Soatto 2020 Vu el al.) 2019). This setup, however, assumes access unlabelled test distribution at training time, whereas in the generalization setting considered here, only one test sample is accessible at inference time, and no knowledge between the subsequent test samples must be shared.
In this work, we study the generalization problem of semantic segmentation from synthetic data (Richter et al. 2016)] Ros et al. 2016) through the lens of adaptation. Instead of modifying the model architecture (Pan et al. (018)) or the training process (Chen et al.) 20200] 2021 Yue et al. 2019).
we enhance the inference procedure with orthogonal techniques inspired by domain adaptation methods (Araslanov and Roth 202T Li et al., 20176). The first mechanism leverages normalization
layers (Toffe and Szegedy 2015) modern convolutional neural networks (CNNs) (He al. 2016) by integrating feature statistics of the test sample. Expanding upon previously attained conclusions for the image classification task (Schneider al., 2020), we find that this strategy not only improves the segmentation accuracy, but also the calibration quality of the prediction confidence. Our second contribution is self-supervised loss, allowing the model to adapt to a single test sample with a few parameter updates. Thirdly, we set up extensive empirical study following a rigorous evaluation protocol, allowing us to establish that the two techniques are complementary. Our study of this one-sample adaptation process reveals consistent improvement in cat-oo-distribuute robustness over the baseline in all benchmark scenarios, and yields a new state of the art in segmentation accuracy, substantially surpassing that of previous work in virtually all considered settings.

---

Under review as a conference paper at ICLR 2022
2 RELATED WORK
Our work contributes to recent research on generalization of semantic segmentation models, and relates to the studies on feature normalization (Pan et al.) 2011:: [Shhneidrr et al. 2020) and test- training (Sun et al. 2020). While the focus in previous investigations was the training strategy (Yue) et al., 2019) and model design (Pan et al. 22088), we exclusively study the test-time inference process here. Yue et al. 2 augmented the synthetic training data by transferring style from real images. Assuming access classification model trained on real images, [Chen et al. (2020) regularize the training on synthetic data by ensuring feature proximity of the two models via distillation, and seek layer-specific learning rates for improved generalization. Advancing the distillation technique, Chen et al. (2021) devise a contrastive loss that facilitates model invariance to standard image augmentations, Pan et al. (2018) heuristically add instance normalization (IN) layers to the network. We remark on two limitations of these works that we address here: First, these methods assume access distribution of real images during training (Chen et al.) 2020 202T Yue al. 2019) (as opposed to only for pre-training of the backbone), or require modification of the network architecture (Pan) al. 2018). Our work requires neither, hence the presented techniques apply even post-hov the already (pre-)trained models to improve their generalization. Second, as we discuss and address in Sec. S.c.5. the evaluation protocol used by previous studies exhibits a number of shortcomings.
Normalization. Batch Normalization (BN; loffe and Szegedy 2015) and other normalization techniques have been increasingly linked to model robustness (Deecke et al., 2019 Huang et al.. Schhneidr et al. 2020, Wang et al.] 2018 Wu and He| 2018). The most commonly used BN, Layer Normalization (LN; Ba et al. (016). and Instance Normalization (IN: [lyanov et al.) (016)) also affect the model's expressive power, which can be further enhanced by their arrangements (Nam and Kim, 2018 Luo et al. 2019). In a domain adaptation setting, [i et al. (2017b) use source-domain statistics during training while replacing them with target-domain statistics during inference. More recently, Schneider et al. (2020) combine the source and target statistics during inference, but the statistics are weighted depending on the number of samples that these statistics aggregate. [ado et al.) (2020) propose using batch statistics during inference from the target domain instead of the training statistics acquired from the source domain. Our comprehensive empirical study complements these results by demonstrating improved generalization of semantic segmentation models.
Test-time training. Test-time training (TTT) refers updating the model parameters also at inference time with self-supervised loss incurred on a single unlabelled test sample. This technique has been recently applied with success for improving the robustness of image classification models (Sun et al. 2020: [Wang et al.] (021a). The design of the self-supervised task is crucial, and the techniques developed for image classification are unsuitable for dense prediction tasks, such as se- mantic segmentation. Nevertheless, recent work explored such losses in domain adaptation scenarios (Araslanov and Roth 202T), and a number of other works exploit commain-specifee knowledge from medical imaging (Varsavsky et al.. 2020) first-person vision (Cai et al. 2020).
Setup comparison. Most of these technically related works (Schneider et al. 2020 Sun et al. 2020: Wang et al., 2021a) focus on the problem of domain adaptation in the context of image classification. They typically assume access to a number of samples (or even all test images) from the target distribution at training time. Our work instead addresses semantic segmentation in the domain generalization setting, which is fundamentally different as it only necessitates a single datum from the test set. In this scenario, simple objectives, such as entropy minimization employed by Tent (Wang et al.) 2021a), improve the bascline accuracy only moderately. By contrast, our one-sample adaptation with pseudo labels accounts for the inherent uncertainty in the predictions, which proves substantially more effective, as the comparison to Tent in Sec. 5.3 reveals. Our task is also different from few-shot learning (e. g., (Finn et al., (017)), where the model may adapt at test time using a small annotated set of image samples. No such annotation is available in our setup; our model adjusts to the test sample in a completely unsupervised fashion, requires neither proxy tasks to update the parameters (Sun et al. 2020) nor any knowledge of the test domain.
3 INSTANCE-ADAPTIVE BATCH NORMALIZATION
Batch Normalization (BN; lofte and Szegedy 2013) has become an inextricable component of modern CNNs (He et al. 2016). Although BN was originally designed for improving training
2

---

Under review as a conference paper at ICLR 2022
convergence, there is now substantial evidence that it plays important role in model robustness Nado et al. 2020), including domain generalization (Pan et al. 2018). Let & E B...C.,WW denote a feature tensor, with C' channels and batch size B, produced by convolutional layer at resolution H x W. We omit the layer indexing, as the following presentation applies to all layers in which BN is applied. At training time, BN first computes the mean and the standard deviation for each of the C channels batch-wise, i. e.
Σ
a²-
E(srest - Mc)2
Pe
BHW Agdule
BHW
2J
The normalized features & follow from applying these statistics,
2r,cjjk (#t.e.3.=--o)) /
Notably, this process differs from the normalization used inference time. At training time, every BN layer maintains a running estimate of μe and oe across the training batches, which we denote here as fe and o.. At test time, is an established practice normalize I. Be and o. in Eq. 2 instead of the test-batch statistics, scheme we refer to as train BN (r-BN).
BN and generalization. In the context of fe and 6. derive from the source data and can be substantially different had they been computed using the target images. This discrepancy is generally known as the covariate shift problem. Domain adaptation methods, which assume access to the (unlabelled) target distribution, often alleviate this issue with a technique typically referred as Adaptive Batch Normalization (AdaBN; [Li al., 2017b). The key idea behind the method is to simply replace the source running statisties with those of the target. Instead of alternating BN layers between the training and testing modes, recent work (Nado et al. 2020) studies prediction-time BN (p-BN), which maintains BN layers in "training mode" also at inference time. That is, p-BN replaces the running statistics from the training time, Aoe and o.. with the statistics [4, and 0g of the current test batch. Such a seemingly innocuous change is shown to benefit model robustness for image classification (Nado et al. 2020 Schneider et al. (020).
generalization, the running statistics
In contrast AdaBN and p-BN, which utilize either the whole target distribution or number of samples, our study of model generalization assumes only a single target example to be available the one that our model receives as the input at inference time. A viable alternative AdaBN and p-BN is to compute the statistics per sample, which amounts to replacing BN layers with Instance Normalization (IN) layers (Ulyanov et al., 2016) after model training. However, this may cause another extreme scenario for covariate shift, since such replacement may significantly interfere with the statistics of the activations in the intermediate layers with which the network was trained. Moreover, previous work showed that IN layers hurt the discriminative power of the model and can improve model robustness only in combination with BN (Pan et al., 2018), thereby requiring changes in the model architecture.
Instance-adaptive batch normalization. Following Schneider et al. 2020), we compute weighted average of the running statistics from training source data and the target statistics, which in contrast to Schneider al. (2020) always derives from a single sample in our work. Let (A(5)) and a! denote the running estimate of the mean and the standard deviation at training time on the source data (for feature channel c). Similarly, define n(! and of as the mean and the standard deviation computed from a feature tensor 2(t) E R.C..W coming from single target sample:
Σ
(t) E.
(1)
1
!1)
(3)
HW
HW
j.k
At inference time, we compute the new mean and standard deviation, M6 and oe as follows:
p = =qplt + +1 (1- a)(
o(-)²= ao!!!? (1) +11
We then use m!()) and of in place of pe and oe in Eq. to normalize the features. Note that this does not affect the behaviour of the BN layers at training time and applies only at test time. Since this approach combines the inductive bias coming in the form of the rumning statistics from the source domain with statistics extracted from a single test instance, we refer to this technique as Instance-adaptive Batch Normalization (laBN).
3

---

Under review as a conference paper at ICLR 2022
Original Image
Lngits
Irassormatioon
CNN
Loss
Augmented Images
Softmax Probabilities
Ignore Mask paeewise threshold per class
Pscudo Ground Truth
Sofumax Probabilities
ingut
per class
Backpropagation
Figure 1: Overview of the one-sample adaptation process. Based on single test sample, we create batch of images by augmenting the original input with multi-scale versions. Furthermore, we add horizontally flipped and grayscaled version for every scale, In order to project the resulting output from every version back the original image, apply the respective inverse affine transformation to every prediction. Afterwards, we average the predicted softmax probabilities and create a pseudo label using dass-dependent confidence threshold. We update the model parameters by minimizing the cross-entropy loss w. x I. the pseudo label and repeat this process for a small number of iterations, Nr, before producing the final prediction. The updated model then discarded.
Setting a in Eq. (4) defaults the established procedure, z-BN, and uses only the running statistics from training on the source domain at inference time. Conversely, a corresponds to Instance Normalization (Ulyanov al. 2016) or, equivalently, the p-BN strategy (Nado et al. 2020) with a batch size of 1. This is in contrast to (Nado et al. 2020), where (t) and de averaged over batch of target images, whereas rely single test sample in this work. Our experiments Sec. [5. expand upon previous analyses (Schneider et al. 2020) with an extensive emperical study of this normalization strategy for semantic segmentation.
Discussion.
Note that Batch Instance Normalization (Nam and Kim, 2018) used a similar weighting approach only for model
for adaptive stylization. However, was training parameter, whereas selection using a validation set.
In Sec. [] we empirically verify that IaBN yields consistent boost of the segmentation accuracy in the out-of-distribution scenario. Perhaps more surprisingly, we also find significant improvements in model calibration (in terms of the expected calibration error, ECE (Naeini et al. 2013)). We leverage this further in devising a self-supervised approach for test-time training of the segmentation model.
4 LEARNING FROM A SINGLE SAMPLE
So far, we assumed that the parameters of the segmentation model remain fixed at inference time. However, both from the perspective of adaptive systems and their biological counterparts, this assumption seems implausible (Thrun 1998; Widmer and Kubat 1996). Here, we allow the model to update its parameters. Note that setup is distinet from the domain adaptation scenario, in contrast to Wang et al. (2021a), since we discard the updated parameters when processing the next sample.
Our approach, visualized in Fig. 1 uses data augmentation images for each test sample. Based on the original test image, we first create a set of augmented images by multi-scaling, horizontal flipping, and grayscaling. These augmented images are used to form a mini-batch, which is fed through the CNN. We project the produced softmax probabilities from the model back the original pixels using the inverse affine transformations, and denote the result as Mi..: for every sample in the mini-batch. This allows the model to have multiple predictions for one pixel. We then compute the mean m of these softmax probabilities along the mini-batch dimension i for class
method to create mini-batches of
1 N Σ Mi.ejjk
meJik
(5)
Using hyperparameter E 1), we compute a threshold value t, from the maximum probability of every class yield class-dependent threshold f.:
1/ max(Mena))
(6)
4

---

Under review as a conference paper at ICLR 2022
We finally extract the class oj with the highest probability for every pixel by a simple arg max operation:
Si. arg max(m..j..).
We ignore low-confidence predictions using our class-dependent threshold bc- Specifically, all pixels with softmax probability below the threshold are set an ignore label, while the remaining pixels use the dominant class of as the pseudo label j.
if max(m...k)) 2 tej,
Uj,b=
ignore,
else.
This generated pseudo ground truth u for the original test image is used to fine-tune the model for N₂ iterations using the cross-entropy loss and gradient descent. We determine all hyperparameters, i. e. resolution of the scales, threshold vb, number of iterations N₁. and learning rate 7/, based on a development dataset. After the one-sample adaptation process, we produce single final prediction using the updated model weights. To process the next test sample, we reset these weights to their initial value, hence the model obtains no knowledge about the complete target data distribution.
5 EXPERIMENTS
Previous studies (Chnn et al. 2020 2021 Pan et al., 2018 Yue et al. 2019) on domain generalization for semantic segmentation used a number of evaluation schemes. For example, Chen et al. (2020) 202T): Pan et al. (018) used only a single target domain, Cityscapes (Cordts et al. (0166). for testing. However, like other research datasets, Cityscapes (Cordts et al., 2016) is carefully curated (e. g., the used for the image capture), hence only partially represents the visual diversity of the world. A more comprehensive approach by Yue et al. (2019) considered number of target domains and used the average accuracy across those as a generalization metric. However, the model was separately selected for every target domain based on a different validation set. As a result, the average accuracy across the tested domains represents the expected accuracy of model ensemble. In this work, in contrast, we aim to improve the accuracy of single model.
same camera hardware and country
To that end, we propose a revised evaluation protocol to quantify the generalization ability of single model. We consider practical scenario in which supplier prepares a model for a consumer without a-priori knowledge on where this model may be deployed. On the supplier's side, we assume access to two data distributions for model training and validation, the source data and the development set. We assess the generalization ability of the model yielded by the validation process on three qualitatively distinct target sets. The average accuracy across these sets provides an estimate of the expected model accuracy for its out-of-distribution deployment on the consumer's s side. Next, we concretize the datasets used in this study, limited to traffic scenes for compatibility with previous work (Chen et al.) 2021 Pn et al. 2018: Yue et al. 2019). and detail the dataset specifics in Appendix FF
Source data. We train model on the training split of two synthetic datasets (mutually exclusive) with low-cost ground truth annotation: GTA (Richter et al. (016) and SYNTHIA (Ros et al. 2016). Importantly, these datasets exhibit visual discrepancy (i. e. domain shift) w. the real imagery, from which our model needs to generalize.
Development set. For model selection and hyperparameter tuning, we use the validation set of WildDash (Zndell et al. 2018). In our scenario, the development set is understood to be of limited quantity, owing its more costly annotation compared to the source data. In contrast the training set, however, it bears closer visual resemblance to the potential target domains.
Multi-target evaluation. Following model selection, we evaluate the single model on three target domains comprising the validation sets from three datasets of street scenes: Cityscapes (Cordts et al.) 2016), BDD (Yu et al.) 2020), and IDD (Varma et al., 2019). The choice of these test domains stems from a number of considerations, such as the geographic origin of the scenes (Cityscapes, BDD, and IDD were collected in Germany, North America, and India, respectively). Geographic distinction well as substantial differences in data acquisition (e. g., camera properties) of these datasets bring together an assortment of challenges for the segmentation model at test time. Since the deployment site of our model is unknown, we assume uniform prior over the target domains as test distribution. Under this assumption, a simple average of the mean segmentation accuracy across target domains yields the expected model accuracy.
5

---

Under review as a conference paper at ICLR 2022
Table (a) Segmentation accuracy using IaBN. We report the mean IoU (%) on three target domains (Cityscapes. BDD. IDD) across both backbones. r-BN denotes train BN (loffe and Szegedy 2013). while p-BN refers to prediction-tin
BN (Nado et al. 2020). (b) ECE (%) for IaBN and MC- Dropour
(Gal and Ghahramani 2016). We report scores for three target domains (Cityscapes, BDD, IDD)
across both backbones. We trained the networks on GTA in both cases (cf. supplemental material for
results with SYNTHIA training).
(a)
(h)
IoU (%, 1)
ECE (%, 4)
Method
Method
CS
BDD
IDD
Mean
CS
BDD
IDD
ResNet-50
ResNet-50 (Baseline)
37.28
35.61
27.73
w/ r-BN
30.95
28.52
32.78
30.75
w/ IaBN
30.57
30.94
26.90
w/ p-BB
37.71
31.67
30.85
33.41
w/ MC-Dropout
30.29
29.80
24.17
w/ IaBN (ours)
37.54
32.79
34.21
34.85
w/ MC-Dropout IaBN
25.50
27.36
22.62
ResNet-101
ResNet-101 (Baseline)
35.24
33.74
27.28
w/ B-N
32.90
32.54
30.36
31.93
w/ IaBN
26.12
28.89
23.98
w/ p-BN
39.88
34.30
33.05
35.74
w/ MC-Dropout
31.30
29.95
25.15
w/ IaB (ours)
42.17
35.40
33.52
37.03
w/ MC-Dropout + IaBN
24.44
28.68
23.32
To compare to previous works, we also evaluate on Mapillary (Neuhold al. 2017). Mapillary does not publicly disclose the geographic origins of individual samples, hence is unsuitable to identify potential location bias acquired by the model from the training data. This possible in our proposed evaluation protocol, since the geographic locations from Cityscapes, BDD, and IDD do not overlap.
Implementation details. We implement our framework in PoTohh (Paszke et al. 2019). Following
(Pan et al. 2018), our baseline model is DeepLabv I (Chen et al. 2015) without CRF post- post-processing, but the reported results also generalize to more advanced architectures (see Appendix D We use ResNet-50 and ResNet-101 (He et al. 2016) trainrrined on ImageNet (Deng et al.) 2009) as backbone. We minimize the cross-entropy loss with an SGD optimizer and a learning rate of 0.005, decayed polynomially with the power set to 0.9. All models are trained on the source domains for 50 epochs with batch size, momentum, and weight decay set to 4, 0.9, and 0.0001, respectively. For data augmentation, we compute crops of random size (0.08 to 1.0) of the original image size, apply random aspect ratio (3/4 to 4/3) to the crop, and then resize the result to 512 x 512 pixels. Furthermore, we use random horizontal flipping, color jitter, random blur, and grayscaling. We train our models with SyncBN (Paszke et al.) 2019) on two NVIDIA GeForce RTX 2080 GPUs.
5.1 EVALUATING IABN
For both source domains (GTA, SYNTHIA) in combination with all main target domains (Cityscapes, BDD, IDD), we investigate the influence of a on the IoU. Setting an optimal a for every target domain infeasible in domain generalization as the target domain during inference unknown. Instead, we choose the optimal in steps of 0.1 based on the IoU on the development set of WildDash. For the ResNet-50 backbone, we attain the highest validation IoU for both training datasets with a = 0.1 (see Fig. in Appendix(B). Fixing this optimal domains. Table reports the segmentation accuracy with the optimal a. For comparison, we also test our models with the two boundary values of O and = corresponding to 1-BN and p-BN (cf. Sec.B). respectively. In Table ([(a) we report IoU scores for both backbones on generalization from GTA to Cityscapes, BDD, and IDD and compare the accuracy of the target domains with 1BN and p-BN. Remarkably, IaBN improves the mean IoU not only of the 1-BN baseline (e. g., by 4. 1% IoU with Res,et-50), which represents an established evaluation mode, but also over the more recent p-BN (Nado et al. 2020). This improvement is consistent across the board, i. e. irrespective of the backbone architecture and the target domain tested. Furthermore, we found that the calibration of our models, in terms of the expected calibration error (ECE; Naeini et al. 2013). also improves. As shown in Table Tbb), not only does IaBN substantially enhance the baseline, but is even competitive with the commonly used MC-Dropout method (Gal and Ghahramani 2016). Rather surprisingly. laBN exhibits a complementary effect with MC-Dropout: the calibration of the predictions improves even further when both methods are used jointly.
proceed with evaluating our model on the target
Our code and pre-trained models will he made publicly available under the Apache License.

---

Under review as a conference paper at ICLR 2022
Table 3: Mean loU (%) with TTA (Simonyan and Zisserman 2015) and our Seg-TTT. We report scores across both source domains (GTA, SYNTHIA) and three target domains (Cityscapes, BDD, IDD) for both backbones.
Source:
GTA
Source: SYNTHIA
Method
CS
BDD
IDD
Mean
CS
BDD
IDD
Mean
ResNet-50 (w/ IaBN)
37.54
32.79
34.21
34.85
36.14
26.66
26.37
29.72
TTA
42.56
37.72
37.98
39.42
39.67
32.10
30,46
34.08
w/ Seg-TTT (ours)
45.13
39.61
40.32
41.69
41.60
33.35
31.22
35.39
ResNet-101 01 (w/ IaBN)
42.17
35.40
33.52
37.03
38.01
28.66
27.28
31.32
w/ TTA
44.37
38.49
38.35
40.40
39,91
32.68
30.04
34.21
Seg-TTT TTT (ours)
46.99
40.21
40.56
42.59
42.32
33.27
31.40
35.66
Table 2: Mean ToU (%) comparison of laBN to
Comparison to other related work. We addi- tionally compare IaBN to alternative normaliza- tion strategies proposed in the literature: Batch- Instance Normalization (BIN; Nam and Kim 2018) and Switchable Normalization (SN: Luo et al. 2019). Although both of these techniques may appear technically similar
alternative normalization strategies: SN [uo et al.)
2019) and BIN (Nam and Kim 2018).
Method
CS
BDD
IDD
Mean
SN BIN
31.75
33,60
31.60
32.32
IaBN, these approaches were developed for different purposes. SN was only shown to improve in- domain accuracy, while BIN tackles domain adaptation for image classification. Our work studies domain generalization. Furthermore, both methods modify the model architecture before training, while IaBN works with any pretrained seman- tic segmentation model. We implemented both in our segmentation model based on the ResNet-50 backbone. We trained these approaches on GTA in an identical setup as IaBN. From results in Table 2 we observe that IaBN outperforms both BIN and SN by a significant margin in terms of mean IoU of the target domains.
34.57
32,68
30.22
32.49
laBN (ours)
37.54
32.79
34.21
34.85
5.2 EVALUATING SEG-TTT
In addition comparing our Seg-TTT standard inference, we test our models against Test-Time Augmentation (TTA) (Simonyan and Zisserman (0155) as a stronger baseline. TTA augments the test samples with their flipped and grayscaled version on multiple scales and averages the predictions as the final result. For Seg-TTT, we use horizontal flipping and grayscaling with factor scales of (0.25, 0.5, 0.75) r. the original image resolution. We study the relative importance of these augmentation types in Appendix C Based on the validation set WildDash, we set threshold 1é = 0.7, use N, = 10 iterations and a learning rate 7) = 0.05. We only train the layers conv4_x, conv5_x and the classification head as we did not observe any benefits from updating all model parameters. Furthermore, this reduces runtime due to not backpropagating through the whole network. We investigate this choice as part of the runtime-acene accurrcy trade-off in Sec. 5.3 In Table we show loU scores for both source domains (GTA, SYNTHIA) and three target domains (Cityscapes, BDD, IDD) across both backbones. Even though TTA improves the baseline (e. g., by 3.37% ToU with ResNet- 101 using GTA), our proposed Seg-TTT still outperforms it by a clear and consistent margin of 2. 2.19% IoU average. This observation aligns well with reported ECE scores in Table (b) to demonstrate that Seg-TTT further exploits the calibrated confidence of our predictions to yield reliable pseudo labels for adapting the model for particular test sample.
5.3 COMBINING IABN AND SEG-TTT
Both IaBN and Seg-TTT are orthogonal techniques and can used jointly. Surprisingly, we find the techniques mutually complementary. We combine IaBN and Seg-TTT in our model and compare with ofate-of-the-ar domain generalization methods in Table 4 While most the other methods substantially
report their results weaker baselines, show consistent improvements even stronger baseline. Our single model with IaBN and Seg-TTT even outperforms the model ensemble approach of Yue et al. (2019) most benchmarks (e. g., by 13.37% and 9.44% GTA to Mapillary
7

---

Under review as a conference paper at ICLR 2022
Table 4: Mean IoU (%) comparison state-of-the-art domain generalization methods for both source domains (GTA, SYNTHIA) as well as three target domains (Cityscapes, Mapillary, BDD). We compare to IBN-Net (Pan et al. 2018), Yue et al. (2019), ASG (Chen al. (020) and CSG (Chen al. 2021). In-domain training to obtain the upper bounds uses our baseline DeepLaby and follows
same schedule as with the synthetic datasets. (+) and (1) denote the use of FCN (Long et al.
2013) and DeepLabv2 (Chen et al. 2018a) architectures, respectively.
Backbone: ResNet-50
Backbone: ResNet- 101
Method
CS
Mapillary
BDD
CS
Mapillary
BDD
In-domain Bound
71.23
58.39
58.53
73.84
62.81
61.19
No Adapt
22.17
17.47
IBN-Net
29.64
No Adapt
32.45 14.97
25,66 34.12
26.73
33.56
28.33
27.76
18.46
15.41 32.14
98.97 42.53
19.72 38.05
110.96
Yue et al.
2019
37.42
38.72
No Adapt
25.88 13.77
29.63
GTA
13.16 32.79
ASG
29.65
No Adapt
25.88
29.63
35.27 19.39
19.25 38.88
CSG
No Adapt
30.95
34.56
28.52
32.90,
36.00
32.54
45.13 44.13
12.93 47.49
11.09
144.09 46.99
111.49
17.67
Ours
39.61
47.49
40.21
No Adapt
28.36
27.24
25.16
29.67
28.73
25.64
SYNTHIA
17.29 35.65
15.50
16.37
17.91 37.58
15.39 34.12
18.70
Yue et al. 2019
32.74
31.53
34.34
No Adapt
31.83
33.41
24.30
37.25
36.84
29.32
41.60 19.77
17.80 41.21
19.05 33.35
15.07 42.32
4.36 41.20
13.95
Ours
33.27
45
TTA
Ensemble (TTA).
ToU (%)
40
IaBN
Ensemble (IaBN)
Seg-TTT TT- (conv4 x + conv5 x Head)
35
Ensemble (Bascline)
Seg-TTT TTT (conv5 x Head)
Baseline
Seg-TTT (Head)
30
0
3
6
9
10
Runtime (s)
Figure 2: Runtine-accuracy comparison GTA -> Cityscapes generalization. The curves trace Seg-TTT iterations, e. the first point corresponds to N₂ = 1, while the last shows N₂ = 10. While Seg-TTT increases the inference time of the baseline and TTA for the sake of improved accuracy, it is still more efficient and accurate than model ensembles of 10 networks. The choice of the layers for Seg-TTT updates (the naming follows He et al. (2016) further provides favorable runtime-accuracy trade-off. Runtimes are computed on single NVIDIA GeForce RTX 2080 GPU.
with ResNet-50 and 101, respectively). Recall that ASG Chen et al. 2020) and CSG (Chen al. 202T) (as well as Yue et (2019) require access distribution of real images for training, while IBN-Net (Pan et al.) 2018) modifies the model architecture. Our approach requires neither, alters only the inference procedure, yet outperforms these methods on most benchmarks substantially. As in the previous discussion, the improvement over the baseline is consistent, regardless of backbone architecture or source data.
Runtime-accurace trade-off. We investigate the influence of the number of iterations required to adapt to a single sample during Seg-TTT. Fig. plots IoU scores for Cityscapes using the ResNet-50 backbone trained GTA (Appendix prrovideeep further numerical comparison). As widely adopted baseline, we also train a model ensemble comprising 10 DeepLabvl networks (as in Seg-TTTI TTT), initialized with random seed (Hansen and Salamon 1990). Note that TTA, Seg-TTT, and the ensemble use IaBN for fair comparison, and we also test the ensemble with TTA. We observe that although Seg-TTT increases the accuracy of the baselines the expense of higher test-time latency,
8

---

Under review as a conference paper at ICLR 2022
GTA + Cityscapes
GTA - BDD
GTA - Mapillary
GTA 7 IDD
Image
Ground Truth
NT
Baseline
Ours
Figure 3: Qualitative semantic segmentation results for the generalization from GTA to Cityscapes, BDD, Mapillary, and IDD for the ResNet-50 backbone. The input image, ground truth, prediction of the baseline model, and prediction of the proposed combination IaBN and Seg-TTT are shown.
it is still more efficient and more accurate than the model ensembles. Furthermore, Seg-TTT offers practical accuracy-
trade-off by means of using fewer update iterations, or updating fewer upper network layers. While the top-accuracy variant of Seg-TTT may not be suitable for real-time applications yet, it may still be valuable in other important domains, such as medical imaging, where high accuracy is desirable even at the cost of increased latency. For real-time needs, IaBN alone boosts the accuracy of the baseline significantly without any computational overhead.
Comparison to Tent (Wang al., 2021a). Like Seg-TTT, Tent also relies test-time training. However, different from constructing the pseudo labels based on well-calibrated predictions in Seg-TTT, Tent simply minimizes the entropy of single-scale prediction. Tent also limits the adaptation to updating only the BN parameters, whereas our Seg- extends this process to convolutional layers. To demonstrate these advantages, we trained HRNet-W (Wang et al.) (02Ib) on GTA and compare the loU on Cityscapes to the equivalent configuration of Tent. While Tent reaches 36.4% with 10 update iterations by adapting the model to single image, IaBN alone already outperforms it substantially with a single forward pass (40.0%), and reaches 44.1% with 10 Seg-TTT TTT update iterations.
Qualitative results. In Fig. 3 we visualize qualitative segmentation results from our joint model with laBN and Seg-TTT for generalization from GTA Cityscapes, BDD, Mapillary, and IDD. We observe a clearly perceivable improvement over the baseline, especially in terms of consistency w. x the image boundaries. Appendix G illustrates further results and offers a discussion of failure cases.
CONCLUSION
We presented a study of an adaptive inference process for improving out-of-distributtin robustness of semantic segmentation models. The accuracy improvement demonstrated in the study surprisingly substantial, despite no changes to the training process or the model architecture, unlike in previous works (Chen et al. 2020 Yue et al. (011).. Considering the simplicity of the studied approach, yet its significant empirical benefits, we hope that such test-time adaptive strategies can inspire follow-up work on improving
generalization of models in other research subfields, potentially extending to other application domains and dense prediction tasks, such panoptic segmentation, or monocular depth prediction. In future work, we are excited to explore more effective self-supervised loss functions, well as efficient test-time adjustments the model parameters.

---

Under review as a conference paper at ICLR 2022
Ethics Statement. Although increased -uttrf--ditributiinn robustness be potentially life-saving in self-driving scenarios, semantic segmentation models can be also deployed with malicious intent, such as unauthorized surveillance. However, semantic segmentation models are still subject to fundamental research with generally low levels of technology readiness (1 to 3) to date and cannot discriminate the object level (in contrast to, 8-, instance segmentation). These reasons make such unwelcome deployment unlikely without any expert knowledge that we commit not to provide. Moreover, the accuracy of semantic segmentation approaches, while impressive, has not yet reached levels where these methods should be uncritically deployed.
Reproducibility Statement. To facilitate reproducibility. we provide our implementation to the reviewers, and commit to releasing it publicly upon acceptance. We have also discussed even trivial, crucially useful training details of our baseline in-depth Appendix A and the start of Sec.
REFERENCES
N. Araslanov and S. Roth, Self-supervised augmentation consistency for adapting semantic segmentation. In CVPR, pages 53384-13399 2021.
M. Arjovsky, L. Bottou, Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv:1907.02899 [stat.M),, 2019.
Ba, R. Kiros, and G. Hinton. Layer normalization. arXiv:1607.06455 [stat.ML). 2016.
S. Ben- Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan. A theory of learning from different domains. Mach, Learn., (9(1-2):151 (2111)::511155 2010.
S. Bickel, M. Brückner, and T. Scheffer. Discriminative learning under covariate shift. JMLR, 10:2137-2155, 2009.
M. Cai, F. Lu, and Y. Sato, Generalizing hand segmentation in egocentric videos with uncertainty guided model adaptation. In CVPR, pages 14392- 14401, 2020.
L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell., 4044))834-848, 2018a.
L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Semantic image segmentation with deep convolutional and fully connected CRFs. In ICLR, 2015.
L.-C. Chen, Y. Zhu, G. Papandreou. F. Schroff, and H. Adam. Encoder-decode with atrous separable convolution for semantic image segmentation. In ECCV, volume 7, pages 833 151, 2018b.
W. Chen, Z. Yu, Z. Wang, and A. Anandkumar. Automated synthetic-to-real generalization. In ICML, pages 1746-1756, 2020.
W. Chen, Z. Yu, S. D. Mello, S. Liu, J. M. Alvarez, Z. Wang, and A. Anandkumar. Contrastive syn-to-real generalization. In ICLR, 2021.
Choi, S. Jung, H. Yun, J. T. Kim, S. Kim, and J. Choo. Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening. In CVPR, pages 11580-1 1590, 2021.
M. Cordts, M. Omran, S. Ramos, Scharwächter, M. Enzweiler, R. Benenson, Franke, S. Roth, and B. Schiele. The Cityscapes dataset for semantic urban scene understanding. In CVPR, pages 3213-3223, 2016.
L. Deecke, Murray, and H. Bilen. Mode normalization. In ICLR, 2019.
J. Deng, W. Dong, Socher, L.-J. Li, K. Li, and F-F.. Li. ImageNet: large-scale hierarchical image database. CVPR, pages 248-255, 2009.
C. Finn, Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, volume 70, pages 1166-1135 2017.
Y. Gal and Z. Ghahramani. Dropout Bayesian approximation: Representing model uncertainty in deep learning. In ICML, pages 1050-1090 1059, 2016.
Y. Ganin, F. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. S. Lempitsky. anversaadveraarial training of neural networks. JMI.R, 17:59:1-59:35, 2016.
10

---

Under review as a conference paper at ICLR 2022
L. Gulrajani and D. Lopez-Paz. In search of lost domain generalization. arXiv:2007.01434 [cs.LG]. 2020.
I. K. Hansen and P. Salamon. Neural network ensembles. IEEE Trans. Pattern Anal. Mach. Intell., 12(10): 993-1001, 1990.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
J. Huang, D. Guan, Xiao, and S. Lu. Fsdr: Frequency space domain randomization for domain generalization. In CVPR, pages 6891-6902. 2021.
Huang, Y. Zhou, F. Zhu, Liu, and L. Shao. Iterative normalization: Beyond standardization towards efficient whitening. CVPR, pages 4874 4883, 2019.
loffe Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, pages 448-456, 2015.
Li, Yang, Y.-Z. Song, and M. Hospedales. Deeper, broader and artier domain generalization. ICCV, pages 5542-5550, 2017a.
Y. Li, N. Wang, J. Shi, J. Liu, and X. Hou. Revisiting batch normalization for practical domain adaptation. In ICLR, 2017b.
J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR, pages 3431-3440, 2015.
M. Long, H. Zhu, J. Wang, and M. I. Jordan. Unsupervised domain adaptation with residual transfer networks. In NewrlPS, pages 336-144, 2016.
P. Luo, J. Ren, Z. Peng, R. Zhang, and J. Li. Differentiable teoming-to-normaaiie via switchable normalization. In ICLR, 2019.
Z. Nado, S. Padhy, D. Sculley, A. D' Amour, B. Lakshminarayanan, and J. Snock. Evaluating prediction-time batch normalization for robustness under covariate shift. urXiv:2006.10%6 [cx.LG], 2020.
M. Naeini, G. F. Cooper, and M. Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. AAAI, pages 2901-2907, 2015.
H. Nam and H. Kim. Batch-instance normalization for adaptively style-invariant neural networks. In NeurlPS, pages 2563-2572, 2018.
Neuhold, Ollmann, Rota Bulò, and Kontschieder. Mapillary Vistas dataset for semantic understanding of street scenes. ICCV, pages 4990-4999, 2017.
X. Pan, Luo, J. Shi, and Tang. Two once: Enhancing learning and generalization capacities via IBN-Net. In ECCV, volume 4, pages 484 500, 2018.
A. Paszke, Gross, F. Massa, A. Lerer, Bradbury, G. Chanan, Killeen, Z. Lin, N. Gimelshein, L. Antiga, ct al. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, pages 8024-8035, 2019.
S. R. Richter, V. Vineet, S. Roth, and V. Koltun. Playing for data: Ground truth from computer games. In ECCV, volume 2, pages 102-112 118, 2016.
G. Ros, L. Sellart. J. Materzynska, D. Vazquez, and A. M. Lopez. The SYNTHIA dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR, pages 3234-3243, 2016.
S. Schneider, E. Rusak, L. Eck, O. Bringmann, W. Brendel, and M. Bethge. Improving robustness against common corruptions by covariate shift adaptation. In NeurlPS, volume 33, pages 11539-11551, 2020.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Y. Sun, X. Wang, Z. Liu, Miller, A. Efros, and M. Hardt. Test-time training with self-supervision for generalization under distribution shifts. ICML,, pages 9229-9248, 2020.
Thrun. Lifelong learning algorithms. Learning to Learn, pages 181 209. Springer, 1998.
A. Torralaa and A. A. Efros. Unbiased look at dataset bias. In CVPR, pages 1 2011.
11

---

Under review as a conference paper at ICLR 2022
D. Ulyanov, A. Vedaldi, and V. Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv:1607.0002 00020002700222222 CV,, 2016.
G. Varma, A. Subramanian, A. M. Namboodiri, M. Chandraker, and C. V. Jawahar. IDD: A dataset for exploring problems autonomous navigation in unconstrained environments. WACV, pages 1743-1751, 2019.
Varsavsky, M. Orbes-Arteaga, C. H. Sudre, M. S. Graham, P. Nachev, and M. J. Cardoso. Test-time unsupervised domain adaptation. In MICCAI, pages 428 436, 2020.
R. Volpi, H. Namkoong. O. Sener, J. C. Duchi, V. Murino, and S. Savarese. Generalizing to unseen domains via adversarial data augmentation. NeurlPS, pages 5339-5349. 2018.
T.HH. Vu, H. Jain, M. Bucher, M. Cord, and P. Pérez. Advent: Adversarial entropy minimization for domain adaptation semantic segmentation. In CVPR, pages 2517-2526, 2019.
D. Wang, E. Shelhamer. S. Liu, B. Olshausen, and Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021a.
G. Wang, J. Peng. P. Luo, X. Wang, and L. Lin. Kalman normalization: Normalizing internal representations across network layers. NewrIPS, pages 21-31, 2018.
J. Wang, Sun, T. Cheng, B. Jiang, C. Deng. Y. Zhao, D. Liu, Y. Mu, M. Tan, X. Wang, W. Liu, and B. Xiao. Deep high-resolution representation learning visual recognition, JEEE Trans, Pattern Anal. Mach. Intell., 43(10):3349 -364, 2021b.
Widmer and Kubat. Learning in the presence concept drift and hidden contexts, Mach. Learn., 23(1): 69-101, 1996.
Wu K. He. Group normalization. In ECCV, pages 3-19, 2018.
Xie, Z. Zheng. L. Chen, and C. Chen. Leamning semantic representations for unsupervised domain adaptation. ICML, pages 5423-5432, 2018.
Y. Yang and S. Soatto. FDA: Fourier domain adaptation for semantic segmentation. In CVPR, pages 4084-4094, 2020.
F. Yu, H. Chen, X. Wang, W. Xian, Chen, F. Liu, V. Madhavan, and T. Darrell. BDD100K: diverse driving dataset for heterogencous multitask learning. In CVPR, pages 2633-2642, 2020.
X. Yue, Y. Zhang, Zhao, A. Sangiovanni- -iincnnniiiineeelli,, K. Keutzer, and B. Gong. Domain randomization and pyramid consistency: Simulation-to-ree generalization without accessing target domain data. In ICCV, pages 2100-2110, 2019.
Zendel, K. Honauer, M. Murschitz, D. Steininger, and G. Domínguez. WildDash Creating hazard-aware benchmarks. In ECCV, volume 6, pages 407-421, 2018.
12

---

Under review as a conference paper at ICLR 2022
A BASELINE
We found number of training details to be crucial for obtaining a highly competitive baseline, i. e. model without our IaBN and Seg-TTT components. Among them is using heavy data augmentation. Recall from Sec. that we used random horizontal flipping, multi-scale cropping with scale range of [0.08, 1.0]. as well as photometric image perturbations color jitter, random blur, and grayscaling. Color jitter, applied with probability 0.5, perturbs image brightness, contrast, and saturation using a factor sampled uniformly from the range [0.7, 1.3). We use different range of [0.9, 1.1] for the hue factor. We randomly blur the image using a Gaussian kernel with the standard deviation sampled from [0.1, 2.0]. Additionally, we convert the image to grayscale with a probability of 0.1. Furthermore, we also found that the polynomial decay schedule we used for the learning rate, as well training for at least 50 epochs (for both GTA and SYNTHIA) are essential to achieve a high baseline accuracy. Note that we only used WildDash as the development set to tune these training details, We also experimented with higher input resolution and larger batch size, but did not observe significant improvement, yet a drastic increase in the computational overhead.
On importance of the baseline. We note that the reported accuracy from previous work in Table 4 exhibits an inconsistency w. F. the choice of the model architectures. In particular, Yue et al. (2019) use an FCN, yet outperform other domain generalization approaches with DeepLaby Pann et al.
2018) and DeepLabv2 (Chen et al.) 2020 2021) considerably, as shown in Table This is a regrettable consequence of inconsistent training schedules used in previous works that proved difficult to reproduce. For example, at the time of submission, Yue et al. (2019) have not released their code despite the p
al. 2018 did not share parts of the code implementing semantic
segmentation These circumstances make reporting the accuracy of the implementation-specie baselines indispensable, which has thus become the standard practice in more recent previous (Chen
et al. 2020 2021) and related works (Gulrajani and Lopez-Paz 2020).
B IABN
Selecting a. Fig. shows a detailed plot of the influence of on the segmentation accuracy, both on the development set of WildDash and on the target domains. We observe that the maximum accuracy on the development set is attained with a 0.1. Clearly, there is no guarantee that value 0.1 is the optimal one for the target domains. However, choosing a based on the development set is in line with the established practice in machine learning: Tuning model hyperparameters is not allowed on the test sets (i. e. Cityscapes, BDD, IDD), but is only possible on the validation set (WildDash). In general, the hyperparameters found to be optimal on the validation set are not guaranteed to remain so on the test set. Nevertheless, our empirical results show consistent improvements over the baselines across all scenarios, despite a having been picked based on the validation dataset.
SYNTHIA as the training set. Due to space constraints, we limited our study of IaBN in the main paper to the scenario of using GTA as the source data (cf. Table D]. Here, we extend this study by training our models SYNTHIA instead. Table reports the segmentation accuracy (in terms of IoU) and the expected calibration error (ECF) for this case. Notably, IaBN can still provide benefits for the expected segmentation accuracy if p-BN (i. e. using target instance normalization statistics) fails to improve over the iBN bascline, as is the case with ResNet-50 in Table the results remain on par with the BBNN baseline even when BBNN is significantly worse than BBN. In regard to calibration quality, the results are consistent with our model trained on GTA (cf. Table Ir: Not only does LaBN improve prediction calibration of the baseline, it again exhibits complementary effect with MC-Dropout. Overall, the combined results from both the model accuracy and the calibration quality of the predictions the ofi-t--dist setting irrespective of the backbone network and specifics of the source data.
that IaBN improves
We use Pillow library httpps//iilloorrretthheddsoo) to implement photometric augmentation. http:://gihuu.co///yye/////////e httph//www..oom/yaye///////saaae htte::/ggtttia.ccom///nangg/nain
13

---

Under review as a conference paper at ICLR 2022
Source: GTA
Source: SYNTHIA
40
30
ToU (%)
20
WildDash
BDD
CS
IDD
10
0.0 0.00011 0.2 0.3 0.4 0.5 0.6 0. 0.8 0.9 1.0
a
Figure 4: Mean loU (%, 7) using IaBN based on the optimal alpha on the development set (WildDash). We report scores for the target domains (Cityscapes, BDD, IDD) for the ResNet-50 backbone after training on GTA (left) and SYNTHIA (right).
Table 5: (a) Segmentation accuracy using IaBN. We report the mean IoU (%) on three target domains (Cityscapes, BDD, IDD) across both backbones. As before, r-BN denotes train BN (loffe and Szegedy 2015), while p-BN refers to prediction time BN (Nado et al. 2020). (b) ECE (%) for IaBN and MC-Dropoul (Gal and Ghahramani 2016). We report scores for three target domains (Cityscapes, BDD, IDD) across both backbones.
We trained the networks SYNTHIA in both cases.
(a)
(b)
IoU (%, f)
ECE (%, 4
Method
Method
CS
BDD
IDD
Mean
CS
BDD
IDD
ResNet-50
ResNet-50 (Baseline)
37.50
43.19
40.11
w/ I-BN
31.83
24.30
24.73
26.95
w/ IaBN
30.96
33.27
36.31
w/ p-BN
33.83
23.36
23.39
26.86
w/ MC-Dropout
34.82
37.30
36.63
IaBN (ours)
36.14
26.66
26.37
29.72
MC-Dropout laBN
30.66
33.06
35.60
ResNet-101
ResNet- 101 (Baselinc)
31.39
33.77
36.56
w/ r-BN
37.25
29.32
27.19
31.25
w/ laBN
30.33
31.83
36.26
w/p-t p-BN
34.58
24.24
22.32
27.05
w/ MC-Dropout
32.73
32.76
34.07
w/ IaBN (ours)
38.01
28.66
27.28
31.32
w/ MC-Dropout laBN
27.71
30.48
32.67
Table The role of the augmentation rype in Seg-TTT. We report mean IoU (%, t) and runtime (ms, 4) for TTA (Simonyan and Zisserman 2015) and our Seg-TTT for the GTA source domain and the Cityscapes target domain for the ResNet-50 backbone.
TTA
Seg-TTT
Method
IoU
Runtime
IoU
Runtime
Baseline
30.95
314
31.47
7135
laBN
37.54
314
39.04
7135
Multiple scales
42.27
749
44.92
7272
Horizontal flipping
38.01
986
39.33
7593
Grayscaling
37.96
908
39.65
7193
Multiple scales horizontal flipping
42.48
1316
44.94
7890
Multiple scales grayscaling
42.28
1202
45.06
7486
Multiple scales horizontal flipping + grayscaling
42.56
1308
45.13
7862
C SEG-TTT
Further implementation details. For Seg- TTT, we only update the model parameters in the layers conv4 x, conv5_ and the classification head for the ResNet-50 backbone. Due to the higher computational cost of the ResNet-101 backbone, we only train the layers conv5 x and the classification head in this case.
14

---

Under review as a conference paper at ICLR 2022
Table 7: Runtime (ms) with TTA (Simonyan and Zisserman 2015) or Seg-TTT. We report the runtime for both ResNet-50 and ResNet-101 101 on three dominant resolutions of 2048 x 1024, 1280 720, and 1920 x 1080, corresponding to the target domains Cityscapes, BDD, and IDD, respectively.
Input resolution
Method
CS
BDD
IDD
Mean
ResNet-50 (single-scale)
314
136
214
221
w/ TTA
1308
713
766
929
w/ Seg-TTT (ours)
7862
3742
5307
5637
ResNet-101 (single-scale)
458
239
252
316
w/ TTA
1519
766
860
1048
Seg-TTT TTT (ours)
9060
4241
6142
6481
The choice of augmentation strategies. We verify the influence of the augmentation type used by Seg-TTT for test-time training. Recall from Sec. Sec. tha we use multiple scales with horizontal flipping and grayscaling to augment one image sample. We compare a flipping-only, scaling-only, and grayscaling-only version of our Seg-TTT to the combination of flipping, grayscaling, and scaling, which we used in the main text. We used ResNet-50 backbone trained on GTA and report the test-time accuracy on Cityscapes in Table We observe significant boost in accuracy in comparison to our IaBN baseline with no augmentations. Furthermore, we show that using multiple scales is more important than flipping for Seg-TTT: Note that the augmentations used do not impact runtime a significant way, since the batch sizes between these setups vary insignificantly; it the backpropagation that dominates the main computational footprint. Varying the number of iterations, trade-off.
as studied in Sec. 5.3 provides a more flexible mechanism for accuracy-
Inference time. Table compares the inference time of our SeT (TTA) and single-scale inference across a range of input resolutions available in the target domains. We obtain these results by running inference improve the runtime estimate, for each dataset we average the inference time over the complete image set. This is account for small deviations in the input resolution (e. g.. IDD mostly contains images of resolution 1920 x 1080, but also has images with resolution 1280 x 720). Since our Seg-TTT uses 10 update iterations, the increase in the inference time w. x. f. TTA is expected. Although such cost may be detrimental for real-time applications, the clear accuracy benefits of Seg- STg (cf. Table 4 may potentially appeal use cases where the importance of the prediction quality outweighs the overhead in the frame rate. Furthermore, we investigated the influence of using the automatic mixed precision module in Py PyTorch and its influence on the inference runtime. While maintaining an identical IoU on Cityscapes using the ResNet-50 backbone, mixed precision achieves runtime of 5746 ms compared to 7862 ms using single precision using N, = 10 for our Seg-TTT. With N, = are even able to decrease the runtime of TTT to 2914 ms.
test-time augmentation
single NVIDIA GeForce RTX 2080 GPU. To
We additionally provide runtime-accuran plots for GTA - BDD and GTA + IDD generalization in Fig. 5 The data supports conclusions drawn on GTA - Cityscapes generalization (cf. Sec. [53 that (1) Seg-TTV provides clear advantages in segmentation accuracy over baselines at a reasonable increase of the inference time; (ii) it is both more accurate and more efficient than model ensembles; and (iii) exhibits a flexible ancuray trade-off by means of varying the number of update iterations and the number of the layers to adjust.
D TESTING IABN AND SEG-TTT WITH OTHER ARCHITECTURES
Our approach generalizes to more recent architectures. We trained four state-of-the-art segmentation models on GTA: DeepLabv3+ (Chen et al. 2018b) with both a ResNet-50 and ResNet-101 101 backbone as well HRNet-W118 18 and HRNet- W48 (Wang et al.) 2021b). Table g reports consistent and substantial improvement of the mean IoU over the baseline, across all these architectures and the target domains.
15

---

Under review as a conference paper at ICLR 2022
40
TTA
Ensemble (TTA)
Ensemble (IaBN)
Yol (%
35
laBN
Ensemble (Baseline)
30
Baseline
Seg- TTT (conv4 + conv5 Head)
Seg-T TTT (conv5 + Head)
Seg- (Head)
25
0
1
2
3
4
6
7
9
10
11
Runtime (s)
40
TTA
Ensemble (TTA)
(%
35
IaBN
Ensemble (IaBN)
JoU
Baseline
Ensemble (Baseline)
30
Seg- TTT (conv4 + conv5 Head)
Seg- TTT (conv5 + Head)
Seg-TTT (Head)
25 0
1
2
3
4
5
6
7
9
10
11
Runtime (s)
Figure 5: Runtime-accurae comparison on GTA - BDD (top) and GTA + IDD (bottom) general- ization. The curves this plot trace Seg-TTT iterations, i. the first point corresponds to N; while the last shows N₂ = 10. While Seg-TTT increases the inference time of the baseline and TTA for the sake of improved accuracy, it is still more efficient and accurate than model ensembles of 10 networks. The choice of the layers for Seg-TTT updates (the naming follows He et al.] further provides a favorable runtime-accuracy trade-off. Runtimes are computed on single NVIDIA GeForce RTX 2080 GPU.
Table 8: Mean loU (%) using our laBN + STT integrated with DeepLabv3+ (Chen et al. 2018b) based on a ResNet-50 and ResNet-101 backbone as well as with HRNei-W18 and HRNet-Wal Wang el al. 2027b). We observe substantial improvements of the segmentation accuracy all three target domains (Cityscapes, BDD, and IDD) after training on GTA.
Target domains
Method
CS
BDD
IDD
Mean
DeepLabyy3+ ResNet-50 (Baseline) w/ laBN + Seg-TTT (ours)
37.51
35.45
37.50
36.82
46.56
43.17
44.07
44.60
DeepLabv3+ ResNet-101 (Baseline) w/ laBN + Seg-TTT (ours)
38.19
37.05
38.22
37.82
48.14
44.52
45.72
46.13
HRNet-Wi W18 (Baseline) w/ LaBN Seg-TTT (ours)
33.08
29.40
32.97
31.82
44.05
38.29
43.78
42.04
HRNet- 448 (Baseline)
34,66
30.85
34.64
33.38
w/ laBN Seg-TTT (ours)
48.82
42.79
43.74
45.12
E COMPARISON TO CONTEMPORANEOUS WORK
In Table 9 compare method the more recent approaches: RobustNet (Choi et al.) 2021 and FSDR (Huang et al., 2021). Choi et al. (202T) disentangle domain-specific and domain-invarte properties from higher-order statistics of the feature representation by using instance selective
16

---

Under review as a conference paper at ICLR 2022
Table 9: Mean IoU (%) comparison to state-of-the-art domain generalization methods for both source domains (GTA, SYNTHIA) as well as three target domains (Citvscapes, Mapillary, BDD). We compare to RobustNet (Choi al. 2021 and FSDR (Huang et al. 2021). ()) and ("*) denote the use
of FCN (Long et al. 2013) and DeepLabv3+ (Chen al. 2018b) architectures, respectively.
Backbone: ResNet-50
Backbone: ResNet-101
Method
CS
Mapillary
BDD
CS
Mapillary
BDD
No Adapt
28.95
28.18
25. 14 10.06
17.63
12.15
Ro
36.58
40.33
35.20
No Adapt
37.75
40.36
33.70
18.36 46.11
19.17 49.53
18.05 41.75
Ours
No Adapt
33.4
27.9
27.3
111.4
115.5
13..9
FSDR
44.8
43.4
41,2
No Adapt
30,95
34.56
28.52
32.90
36.00
32.54
Ours
14.18 45.13
12.99 47.49
39.61
11.009
46.99
14.009
111.49 47.49
17.67 40.21
No Adapt
SYNTHIA
FSDR
40.8
39.6
37.4
No Adapt
31.83
33.41
24.30
37.25
36.84
29.32
19.77
17.80
19.05
15.07
14.36
13.95
Ours
41.60
41.21
33.35
42.32
41.20
33.27
whitening loss. FSDR (Huang et al. 2021) randomizes images in the frequency space by keeping domain-invariant frequency components and varying the domain-sensitive frequency components.
For fair comparison with RobustNet, we trained DeepLabv3+ model with the output stride of 16 in contrast to the output stride of 8 used by default (e. g. in Table 8). We also remark that FSDR used the target domains for hyperparameter tuning and model selection, which gives it a clear advantage
5) Nonetheless, we outperform FSDR in terms of IoU on two out of three target domains using ResNet-101 backbone. Remarkably, our baseline training schedule and data augmentation alone already reach the accuracy of RobustNet without any additional regularization or changes to the model architecture. Our IaBN with Seg-TTT further improve over these results by significant margins (more than 8% IoU) across all tested target domains.
F DATASET DETAILS
GTA. GTA Richteer et al. 2016 is a street view dataset generated semi-automatically from the computer game Grand Theft Auto The dataset consists of 12,403 training images, 6,382 validation images, and 6, 181 testing images of resolution 1914 x 1052 with 19 different semantic classes.
SYNTHIA. We use the SYNTHIA- NANTHIA-RAND-OO MAAD-CCITSSCAPES subset of the synthetic dataset SYNTHIA (Ros et al. (016)) which contains 9,400 images, and has 16 semantic classes in common with GTA. Images have a resolution of 1280 x 760 pixels.
WildDash. This dataset was developed to evaluate models regarding their robustness for driving scenarios under weal-world conditions. It comprises 4256 images of real-world scenes with resolution of 1920 x 1080 pixels.
Cityscapes. Cityscapes (Cordts et al. 2016 is an ego-centric street-scene dataset and contains 5,000 high-resolution images with 2048 x 1024 pixels. It is split into 2,975 train, 500 val, and 1,525 test images with 19 semantic classes being annotated.
BDD. BDD (Yu et al. 2020) is driving video dataset, which also contains semantic labelings with the identical 19 classes as in the other datasets. Images have a resolution of 1280 720 pixels. The training, validation, and test sets contain 7,000, 1,000, and 2,000 images, respectively.
IDD. IDD (Varma et al. 2019) is a dataset for road scene understanding in unstructured environments. It contains 10,003 images annotated with 34 classes even though only evaluate the 19 classes

---

Under review as a conference paper at ICLR 2022
GTA 7 Cityscapes
GTA - BDD
GTA - Mapillary
GTA - IDD
Image
Ground Truth
Baseline
Ours
Figure 6: Qualitative semantic segmentation results for generalization from GTA to Cityscapes, BDD, Mapillary, and IDD for the ResNet-101 backbone.
overlapping with the other datasets. IDD is split into 6,993 training images, 981 validation images, and 2,029 test images.
Mapillary. Annotations from Mapillary (Neuhold et al. 2017 contain 66 object classes; analogously IDD only evaluate the 19 classes overlapping with the other datasets. The dataset split into a training set with 18,000 images and a validation set with 2,000 images with a minimum resolution of 1920 x 1080 pixels.
G QUALITATIVE EXAMPLES
We provide additional qualitative results by running inference three models: ResNet-101 trained on GTA in Fig. 6 ResNet-50 and ResNet-101 trained on SYNTHIA in Figs. [ [ g respectively. Similar observations in Sec. 6.3 (ef. main text), approach exhibits semantic masks with visibly fewer jagged-shaped artifacts than the baseline (e. g., "sidewalk" false positives Fig. G). Models with our IaBN and Seg-TTV TTT may still struggle with cases mislabeling regions with an incorrect, but semantically related class. For example, the model often assigns "sidewalk" the road pixels from BDD and IDD in Figs. and This is expected outcome the erroneous labels are already contained in the pseudo labels of the initial prediction, on which Seg-TTT relies. These failure cases occur more frequently the domain shift between the train and the test distributions is more significant, such as between SYNTHIA and BDD, which can lead to poorly calibrated predictions. Since applying IaBN results in improved calibration (cf. Table(Tp alleviates this issue, and Seg-TTT can cope well with milder domain shift scenarios, result. As examples on Cityscapes and Mapillary Figs. and g show, despite the baseline model exhibiting some degree of this failure mode, inference method visibly rectifies these errors.
homogenous
We additionally ran our inference on video sequences and include the results as part our sup- plementary material. Confirming our previous analysis of the qualitative results (ef. Sec. (.3)). observe that our approach clearly improves the segmentation quality and removes some of the most pathological failure modes of the baseline (e. g., the lower middle part of the frame area).
18

---

Under review as a conference paper at ICLR 2022
SYNTHIA -
Cityscapes
SYNTHIA - BDD
SYNTHIA - Mapillary
SYNTHIA - IDD
Image
Ground Truth
Baseline
Ours
Figure 7: Qualitative semantic segmentation results for generalization from SYNTHIA to Cityscapes,
BDD, Mapillary, and IDD for the ResNet-50 backbone.
SYNTHIA - Cityscapes
SYNTHIA - BDD
SYNTHIA - Mapillary
SYNTHIA + IDD
Image
Ground Truth
Baseline
Ours
Figure 8: Qualitative semantic segmentation results for generalization from SYNTHIA Cityscapes,
BDD, Mapillary, and IDD for the ResNet- 101 backbone.
H FAILURE CASES
Conceptually, if the initial semantic prediction is incorrect and confident (due to miscalibration), likely to end up in the pseudo labels and lead astray the adaptation process. The rightmost columns in Fig. 7 and Fig. X already visualized this failure mode: the baseline largely misclassifies "road" as the "sidewalk" class. We have further extended these examples with Fig. Misguided by incorrect pseudo labels, our test- rest-time adaptation may exacerbate this issue, e. propagate the incorrect label to the areas sharing the same appearance. To gain further insights, we investigated this issue statistically.
19

---

Under review as a conference paper at ICLR 2022
GTA + Cityscapes
GTA - BDD
GTA - Mapillary
GTA - IDD
Image
Ground Truth
Baseline
Ours
Figure 9: Failure cases of semantic segmentation for generalization results from GTA to Cityscapes,
BDD, Mapillary, and IDD for the ResNet-50 backbone.
CS
BDD
IDD
Mapillary
40
Amount of fimages (%)
30
20
10
10 0 10
20
30
40
10 0 10 20 30 40
10 0
10
20
30
40
10 O 10
20
30
40
A (%)
Figure 10: Density distribution of the individual images for generalization from source domain (GTA) target domains (Ciryscapes, BDD, IDD and Mapillary) for the ResNet-50 backbone. We visualize the relative improvement of our test-time inference strategy w.r.t. the baseline in terms of accuracy with A IoU (%).
The histograms in Fig. 10 show relative improvement of our test-time inference strategy the baseline in terms of loU on four validation sets. We conclude that such decrease in segmentation accuracy is actually rather rare: less than 10% of the images, on average, exhibit lower accuracy. In most such cases the accuracy reduces only marginally (by less than 5%), and only a fraction images (less than 1 % average) deteriorate in accuracy by most 10%. The overwhelming majority of the image samples benefit from our test- rest-time adaptive process and increase in accuracy by up to 35% loU.
20

---

Under review as a conference paper at ICLR 2022
1 CODE
We provide an implementation of our approach as part of the supplementary material. The enclosed README provides further details running this code for both training and inference. Upon acceptance, this code will be released to the community under the Apache License. Furthermore, to reproduce the state-of-the-ar results of our models reported in Table we make their snapshots available via an anonymous link provided in the README file.
21


Reviews:
[{"review_id": "sy8Ry-pFUk", "reviewer": "Reviewer_Dswx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n- Test-time training is an interesting concept and the paper spans a nice logical bridge from the first, technical rather minor conribution, that improves calibration of the prediction to leverage areas that are likely predicted correctly to further improve th inference results.\n\n- Interesting, but a bit heuristic, combination of test-time training and pseudo-labeling.\n\n- Good results and broad evaluation.\n\nWeaknesses:\n\n- Inference speed: Test-time training is by definition rather costly and the proposed approach increases runtime by more than an order of magnitude when compared to the baseline. It is unclear how to improve the practicality of the approach.\n\n- When comparing to Wang 2021a it is not clear to me how much improvement actually comes from using multi-scale predictions and augmentations. It would be great to have an experiment along these lines.\n\n- No comparison to (offline) pseudo-labeling approaches (a recent example would be [A]). The overall strategy would also be applicable without test-time training in principle. I.e. one could derive pseudo-labels for the whole target dataset and adapt the classifier once on the full target dataset. While the assumption here is that only a single image from the target domain is available, the setting seems to be rather artificial as a few unlabeled images from the target domain should almost always be available (and test-time training might help on top of such a transfer procedure).\n\nOther questions:\n\n- It would be interesting to see some failure cases (if any). Does test-time training always improve results?\n\n[A] Zou et al., PseudoSeg: Designing Pseudo Labels for Semantic Segmentation, ICLR 2021\n\n", "summary_of_the_paper": "This paper contributes two techniques to improve generalization of semantic segmentation networks. The first technique is an adaptation of the test-time behaviour of batch normalization, where the statistics of the sample under consideration are blended into the training-time statistics of the batch-norm layer. The second technique is to use test time augmentations to first derive a pseudo-labeling from the sample and then to update parts of the weights to maximize the probability of the pseudo-labels. The paper additionally contributes a multi-dataset evaluation procedure and shows favorable performance of the proposed techniques on this evaluation protocol.", "summary_of_the_review": "I think the overall approach is interesting, but ad-hoc in its design. It would be great to see some comparisons to existing pseudo-labeling approaches as I think that the setting with only a single image from the target domain artificial.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fKgU61vajKw", "reviewer": "Reviewer_tfFH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros)\n\nThe paper is well written and easy to understand. The experiments including comparison to baselines and ablation studies seem to be comprehensive.\n\nCons)\n\nThe idea on combining BN and IN has been proposed before. The novelty is limited. In addition, in this work, the hyperparameter, alpha, has to be decided by validation data.\n\nTT-SEG is a simple method for test-time finetuning. Although the effectiveness was shown by the experiments, the novelty seems to be limited.\n\nThe description on baselines such as ASG and CSG in Related Work is too brief to understand the principle difference to the proposed method and among them. \n\nTaking six seconds is too much from practical point of view. As the authors mentioned, the proposed method is unpractical for road scene analysis.\n\nComment)\n\nWhy was ECE used only for Table1(b) without showing MIoU ? Is MIoU also improved by introducing MC-Dropout ?\n ", "summary_of_the_paper": "This paper presents a method on generalization of segmentation from synthetic data to real street scene data. To adapt the model pre-trained with synthetic source domain data such as GTA and SYNTHIA to target domain such Cityscapes, BDD and IDD, the authors proposes Instance-adaptive Batch Normalization (IaBN). In addition, as a method to learn from a single sample, TT-SEG was proposed\nin which a pseudo GT mask is estimated from augmented images with the initial model and the last parts of the network are finetuned with the pseudo GT mask. In the experiments, the proposed method which employs both IaBN and TT-SEG successfully outperformed all the baselines, which indicated the effectiveness of the proposed method.\n\n", "summary_of_the_review": "The novelty of the proposed method is limited, since IaBN is an existing method and the method on one-sample adaptation is straight-forward. Although the effectiveness of the proposed method regarding performance was shown by the experiments, the required processing time was too much. From these points, this paper is regarded as being around the border.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "H07JreMKnr", "reviewer": "Reviewer_YkXo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1.\tThe writing is good, making the paper very easy to understand.\n2.\tBoth the two modules are conceptually simple. \n3.\tExtensive experiments have been performed for evaluation.\n4.\tCode is provided.\n\nCons:\n1.\tMy major concern goes to the novelty of the paper. The proposed normalization technique is incremental. Instance-level normalization is a smart way for test-time adaptation, which has been well studied [1][2][3]. For the test-time training part, it is also hard to claim a unique contribution since it mostly follows a pseudo labeling-based segmentation paradigm. \n\n  [1] Test-time adaptable neural networks for robust medical image segmentation\n  [2] Autoencoder based self-supervised test-time adaptation for medical image analysis\n  [3] Tent: Fully Test-Time Adaptation by Entropy Minimization\n\n2.\tIt is not clear to me the advantage to learn \\alpha using a validation set, rather than online learning during test-time finetuning. \n3.\tThe necessity to use an independent development set for model/parameter selection needs more studies. It is also not clear to me how to select a universe development set that can fit any target domain, or should we find a unique dev set for each target domain.\n4.\tThe model is currently evaluated using relatively lightweight backbones (Res50, HRNet18), and it is not clear whether the method can work well for larger models (e.g., Res101, HRNet48) and lead to consistent improvements.\n", "summary_of_the_paper": "The article proposes two techniques for test-time adaptation. The first is instance-adaptive bn, to combine statistics from source data with each single test sample.  The second is test-time training, based on pseudo labeling of the test sample. The two parts lead to promising results.", "summary_of_the_review": "Overall, the paper has some merits (e.g., good writing, promising results). However, the technical contributions seems to be minor. I am concerning that the paper may be still under the level of ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F_AtFaHN-6L", "reviewer": "Reviewer_FDdf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are strengths of this paper: \n+ The paper is overall well-written and the reviewer can easily follow the whole draft and get the key points of this paper. \n\n+ This paper shows consistent better than previous method. Besides, this paper also analyze two components of this work to demonstrate the effectiveness of their work. According to the experiments  and ablation study using GTA5, Cityscapes, BDD100k, IDD, their method achieves more favorable results than some previous methods including t-BN and p-BN, where their method is a tradeoff between them.  Besides, this paper also shows better results than previous test-time training method. \n\nThe weakness of this paper are as follows:\n- The novelty of this work is the main drawback. The first point instance-adaptive normalization is a trivial solution, which is the tradeoff of previous t-BN and p-BN. Second, there are a lot of work applying pseudo labels in semantic segmentation, such as Naive-students [ECCV2020]. Even though this work leverages pseudo labels for a different application, the technique contribution is not strong.\n\n- Comparison to more previous work [a,b,c] in domain generalization in semantic segmentation is needed. \n[a] RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening, CVPR, 2021.\n[b] FSDR: Frequency Space Domain Randomization for Domain Generalization, CVPR, 2021.\n[c] Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Data, ICCV, 2019.\n\n- In instance-adaptive batch normalization paragraph, the authors use \\head{u}_{c}^{s}, but in Eq (4), the authors use u_{c}^{s}. Need to confirm the notations in the paragraph.\n\n- In the experiments, the paper uses 0.1, 0 and 1 for the alpha in Eq 4, where 0 and 1 are the baseline method. Is the method sensitive to alpha in acheving good results, more analysis is needed. For various target datasets, they have different levels of domain gap, it is unclear how to choose the parameter for \\alpha.\n\n\nQuestion:\n1. What is the motivation to test on the expected calibration error with a combination  of MCDropout ?\n2. How to choose alpha for Eq. 4? \n\n\n", "summary_of_the_paper": "This paper studies an existing problem - domain generalization for semantic segmentation of urban scenes. The technique keys of this work has two main points: instance-adaptive batch normalization and testing-time training using pseudo labels. Instance-adaptive batch normalization aims to bias to the data distribution of individual testing examples, which is a tradeoff of previous t-BN and p-BN. This paper generate pseudo labels for test-time training, which is not quite novel. To evaluate the effectiveness of the proposed method, this paper conducts experiments on GTA5/SYNTHIA -> Cityscapes, BDD100k and IDD, and shows better results.\n", "summary_of_the_review": "This paper presents a new model and pipeline for domain generalization in semantic segmentation. Although this work shows better results than previous work. Lacking of novelty and insights is the main drawback of this work. Besides, the experiments can be improved, that more comparisons with previous work is helpful. Therefore, the reviewer rates this paper marginally below the acceptance threshold. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission presents a technique to improve generalization of urban scenes segmentation.  Based on a pre-trained deep net on synthetic data, the approach aims at adapting statistics on real target domain such as Cityscapes, BDD or IDD datasets using an Instance-adaptive Batch Normalization (IaBN) at test time. Results are reported on several synthetic to real scenarios.

Most of the reviewers were not convinced by the approach and have raised several issues. After rebuttal and discussion, no one really changed her/his mind. The novelty of the proposed method is limited to the use of the existing IaBN in this context except the one-sample adaptation.  Although the proposed method is effective on some benchmarks, the extra processing time may be a significant limitation. Additional comparisons are necessary. We encourage the authors to consider the reviewers feedbacks for future publication.
Decision: Reject