Semantic Self-adaptation: Enhancing Generalization with a Single Sample
Authors: Sherwin Bahmani
Oliver Hahn
Eduard Zamfir
Nikita Araslanov
Daniel Cremers
Stefan Roth
Venue: Accepted by TMLR ()

Abstract: The lack of out-of-domain generalization is a critical weakness of deep networks for semantic segmentation. Previous studies relied on the assumption of a static model, i. e., once the training process is complete, model parameters remain fixed at test time. In this work, we challenge this premise with a self-adaptive approach for semantic segmentation that adjusts the inference process to each input sample. Self-adaptation operates on two levels. First, it fine-tunes the parameters of convolutional layers to the input image using consistency regularization. Second, in Batch Normalization layers, self-adaptation interpolates between the training and the reference distribution derived from a single test sample. Despite both techniques being well known in the literature, their combination sets new state-of-the-art accuracy on synthetic-to-real generalization benchmarks. Our empirical study suggests that self-adaptation may complement the established practice of model regularization at training time for improving deep network generalization to out-of-domain data. Our code and pre-trained models are available at https://github.com/visinf/self-adaptive.

Published in Transactions on Machine Learning Research (07 (07/2023)
Semantic Self-adaptation:
Enhancing Generalization with a Single Sample
Sherwin Bahmani*
Oliver Hahn"
Eduard Zamfir
Nikita Araslanoyf3
Daniel Cremers
Stefan Roth
'TU Darmstadt
Unniversity of Würzburg
"TU Munich "hessian.Al
Reviewed OpenReview: https///paarvveeenr//////ulllllt
Abstract
The lack of out-of-domain generalization a critical weakness of deep networks for semantic segmentation. Previous studies relied on the assumption of static model, i. e., once the training process is complete, model paramcters remain fixed at test time. In this work, we challenge this premise with a self-adaplive approach for semantie segmentation that adjusts the inference process to each input sample. Self-adaptation operates two levels. First, it fine-tunes the parameters of convolutional layers to the input image using consistency regularization. Second, in Batch Normalization layers, selfaddapaation interpolates between the training and the reference distribution derived from a single test. sample. Despite both techniques being well known in the literature, their combination sets new caate-of-thh-arr accuracy on synthetic-to-rcal generalization benchmarks. Our empirical study suggests that self-adaplation may complement the established practice of model regularization at training time for improving deep network generalization to out-of-domain data. Our code and pre-trained models are available at https: http:/ggtthub.ooom disinn/self-adaptiee
1
Introduction
The current state of the art for semantic segmentation (Chen et al. in out-of-distribution robustness, i. e., when the training and testing distributions are different. Numerous studies have investigated this issue, primarily focusing on image classification (Arjovsky et al. 2019: Bickel
2018b Long et al. 2015 lacks direly
et al. 2009 Li et al. 2017a Torralba Efros 2011 Volpi et al. 2018). However, a recent study of existing domain generalization methods (Gulrajani Lopez-Pas 2021 comes to a sobering conclusion: Empirical Risk Minimization (ERM), which makes an d. assumption of the training and testing samples, still highly competitive. This is in stark contrast to the evident advances in the area of domain adaptation, both for image classification (Ben-David et al. 2010
Ganin et al., 2016 Long et al.) 2016 Xie et al. 2018 and
semantic segmentation AArsslanvv & Roth 2021
Vu et al. 2019 Yang & Soatto 2020 This setup, however,
assumees access to an unlabelled test distribution at training time. In contrast, in the generalization setting considered here, only one test sample is accessible at inference time and no knowledge between the subsequent lest sumples must be shared.
In this work, we study the generalization problem of semantic segmentation from synthetic data [Richter et al, 2016; Ros et al.) 2016 through the lens of adaptation. In contrast previous work that modified the model architect.ure (Pan et al. 2018) the training process (Chen el al. 2021 2020 Yue el al. 2019), we revise the standard inference procedure with a technique inspired by domain adaptation methods (Araslanov & Roth 2021 Li et al. 2017b). The technique, that we term self-adaptation, leverages a self-supervised loss, which allows for adapting to a single test sample with a few parameter updates. Complementary to these loss-based updates, self-adaptation integrates feature statistics of the training data with those of the test sample in the Batch Normalization layers (loffe & Szegedy 2015). commonly employed in modern convolutional neural
"Equal contribution; work primarily done while TU Darmstadt.

---

Published in Transactions on Machine Learning Research (07/2023)
networks (CNNs) (He et al. 2016). Expanding upon the previous conclusions in related studies (Schneider et al. 2020), we find that this normalization strategy not only improves the segmentation ion accuracy, but also the calibration quality of the prediction confidence.
In summary, our contributions are the following: (i) We propose a self-adaptive process for model generalization in the context of semantic segmentation, where model inference adjusts to each test sample. (tt We overcome deficiencies in the experimental protocol used in previous studies with a rigorous revision. (ti) Implemented on top of a simple bascline, self-adaptation surpasses previous work and achieves new state-of-the-art segmentation accuracy in muathetic-t--eed generalization.
2 Related work
Our work contributes to recent research on generalization of semantic segmentation models, and relates to studies on feature normalization (Pan et al. 2018 Schneider et al. (020)) and online learning (Sun et al. 2020). While the focus in previous investigations was the training strategy (Yue et al. 2019 and model design (Pan et al. 2018), we exclusively study the test-time inference process here. Yuue et al. (2019) augment the synthetic training data by transferring style from real images. Assuming access to classification model trained on real images, Chen et al. (2020) regularize the training on synthetic data by ensuring feature proximity of the two models via distillation, and seek layer-specific learning rates for improved generalization. Advancing the distillation technique. Chen et al. (2021) devise a contrastive loss that facilitates model invariance to standard image augmentations. Pan el al. (2018) heuristically add instance normalization (IN) layers to the network. More recently, Choi et al. (2021) and Huang et al. (2021) extract innariinn feature statistics by either using an instance-selective whitening loss or frequency-base
domaín randomization. Similarly, Nam ct al. 2022) learn a style-invariani representation by using a causal framework for data generation. Kundu 2021 increase source domain diversity by augmenting single-domain data to virtually simulate
et al. multi-source scenario. [Tang et al. (2021) swap channel-wise statistics in feature normalization layers and learn adapter functions to re-adjust the mean and variance based on the input sample. Lee et al. (2022a) enforces consistency of the output. logits across multiple images (or pixels) of the same class. To improve generalization in federated learning, [Caldarola et al. (2022) train clients locally with sharpness-aware minimization and averaging stochastic weights. However, these methods assume access to a distribution of real images during training (Chen et al., 22200) 2021 Yue et al. 2019) (as opposed to only for pre-training of the backbone), or require modification of the network architecture (Pan et al., (0018). Our work requires neither, hence the presented technique applics even post-hoc to the already (pre-)traince models to improve their generalization. Moreover, as we discuss in Sec. 5] the evaluation protocol used by previous studies exhibits number of shortcomings, which we also address.
Normalization. Batch Normalization (BN: Loffe & Szegody 2015 and other normalization techniques have
been increasingly linked to model robustness (Deecke el al.] 2019 Huang et al. 2019 Schneider et al. 2020
Wang et al. 20118 Wu & He 2018). The most commonly used BN, Layer Normalization (LN; Ba al. 2016).
and Instance Normalization (IN; Ulyanvv et al. 2016) also affect the model's expressive power, which
be
further enhanced by combining these techniques in one architecture (Luo et al. 2019 Nam & Kim 2018 In a
domain adaptation setting, [T et al. (2017b) use source-domain statistics during training while replacing them with target-domain statistics during inference. More recently, Schneider et al. (2020) combine the source and target statistics during inference, but the statistics are weighted depending on the number of samples that these statistics aggregate. Nado et al. (2020) propose using batch statistics during inference from the target. domain instcad of the training statistics acquired from the source domain. Our comprehensive empirical study complements these results by demonstrating improved generalization of semantic segmentation models.
Adapting the model to a test sample. Several examples from previous work update the model parameters at the time of inference. In object tracking. the object detector has the need to adjust to the changing appearance model of the tracked instance (Kalal el. al. 2012). Conditions ional generative models can learn from a single image sample for the task of super-resolution
Glasner et al. 2009) and scene synthesis (Shaham ct al. 2019). More recently, this principle has been used for improving the robustness of image classification models (Sun et al. 2020) Wang et al., [021a] Zhang et al. 2022). The design of the self-supervised task to perform on the test sample is crucial, and the techniques developed for image classification do not al always
2

---

Published in Transactions on Machine Learning Research (07 2022)
Original Image
Semantic Logits
Affine Inverse Transformation
Confidence
Filtering
CNN
Loss
Augmented Images
Aligned
Pseudo
Semantic Maps
Maps
Ground Truth
(Softmaxx)
Backpropagation
Figure 1: Overview of the one-somple adaptation process. We augment single test sample by creating batch of images at multiple scales, each with horizontal flipping and grayscaling. To transform the output of each version back to the original image plane, we apply the corresponding inverse affine transformation to every prediction. After averaging the softmax probabilities, we create a pseudo-label using class-dependent confidence threshold. We update the model parameters by minimizing the cross-entropy loss with respect to the pseudo-label, repeating this process for a small number of iterations (N₂) before producing the final prediction. The updated model is then discarded.
extend dense prediction tasks, such as semantic segmentation considered here. Nevertheless, more suitable alternatives for the self-supervise loss have been recently proposed for domain adaptation (Araslanov &
Roth 2021 and number of other works devised domain-specific approaches for medical imaging (Varsavsyy
ct al. 2020 or first-person vision (Cai et al. 2020). Concurrently to our work, Reddy et al. (2022 enforce
edge consistency during inference for unsupervised test-time adaptation of semantic segmentation methods.
Setup comparison. Most these technically related works (Schneider et al, 2020 Sun et al. 2020 Wang
et al. 2021a) focus on the problem of domain adaptation in the context image classification. They typically assume access to a number of samples (or even all test images) from the target distribution at training time, Our work instead addresses semantic segmentation in the domaín generalization setting, which is different. as it only necessitates single datur from the test set. In this scenario, simple objectives, such as entropy minimization employed by Tent (Wang et al. 2021a), improve the bascline accuracy only moderately. By contrast, our self-adaptation with pseudo-labels accounts for the inherent uncertainty in the predictions, which proves substantially more effective, as the comparison to Tent in Sec. 5.3 reveals. Our task is also different from few-shot learning (e. 9-: Finn et al. 2017), where the model may adapt at test time using a small annotated set of image samples. No such annotation is available in our setup; our model adjusts to the test sample in completely unsupervised fashion, requires neither proxy tasks to update the parameters (Sun) al. 2020 nor any prior knowledge of the test distribution.
3 Self-adaptive learning from a single sample
The traditional assumption at inference time that the parameters of the segmentation model remain fixed. However, jeelf-)adaptiv systems and their biological counterparts provide an example, where they learn specialize the particularities of their environment [Thun 1998 Widmer & Kubat 1996). By analogy, we here allow the segmentation model to update its parameters. Note that setup is distinct from the domain adaptation scenario (c. g., in contrast to Wang et al. 2021a since we discard the updated parameters when processing the next sample in line with the underlying concept of domain generalization.
Our approach, visualized in Fig. I and summarized in Algorithm! [ uses data augmentation as a method to create mini-batches of images for each test sample. Based on the original test image, we first create set of N augmented images by multi-scaling, horizontal flipping, and grayscaling. These augmented images are used to form mini-batch, which is fed through the CNN. We transform the produced softmax probabilities
3

---

Published in Transactions on Machine Learning Research (07 (072))23)
Algorithm 1: Summary of sell-adaptation.
Train segmentation model on source data (best-practics established methodology)
Replace BatchNorm with SaN (cf. Sec. 3.1
Tune hyperparameter a in SaN on validation set (WildDash)
# Inference on any dataset. Initial model parameters: 0g-
foreach test sample do
Obtain 0* by minimizing cross-entropy w.r.t. pseudo-labels in Eq. (4
Predict segmentation for the test sample using 0*
Reset model paramcters to do.
end
from the model back to the original pixels using the inverse affine transformations, and denote the result as Ma.n: for every sample in the mini-batch. This allows the model to have multiple predictions for one pixcl. We then compute the mean mn of these softmax probabilities along the mini-batch dimension for class and pixel (j, the spatial grid as
Mej.k
Mi.jj.k
(1)
Using hyperparameter v E (0,1), 1), we compute a threshold value te from the maximum probability of every class to yield a class-dependent threshold te:
te = 3b max(me..).
(2)
Finally, for every pixel, we extract the class cf with the highest probability by
S arg (...k)....)...
(3)
We ignore low-confidence predictions using our class-dependent threshold fe- Specifically, all pixels with a softmax probability below the threshold are set to an ignore label, while the remaining pixels use the dominant class Gf as the pseudo-label Uz
C if max 2 Eoj
Uj.k
(4)
ignore, otherwise.
The pseudo ground truth for the test image is used to fine-tune the model for N, iterations with gradient descent using the cross-entropy loss. We determine all hyperparameters, i. e., resolution of the scales, threshold v., number of iterations Nr, and learning rate based on a validation dataset. After the self-adaptation process, we produce a single final prediction using the updated model weights. To process the next test sample, we reset. these weights their initial value, hence the model obtains no knowledge about the complete target data distribution.
One natural consideration in this process is the quality of the confidence calibration in the semantic maps, since the arg max operation in Eq. (3) and applying the threshold in Eq. (4) aim to select only the most confident pixel predictions. If the confidence values become miscalibrated (e. g., due to the domain shift), a significant fraction of incorrect pixel labels will end up in the pseudo-mask. Our self-adaptive normalization, which we detail next, mitigates this issue.
3.1 Self-adaptive normalization
Batch Normalization (BN: Toffe & Szegedy 2015) has become an inextricable component of modern CNNs He et al. 2016). Although BN was originally designed for improving training convergence, there is now substantial evidence that it plays an important role model robustness (Nado al.) 2020), including domain generalization (Pan et al.) 2018). Let I B.H.W denote a spatial tensor of size W for an arbitrary feature channel and batch size B, produced by convolutional layer. We omit the layer and channel indexing,
4

---

Published in Transactions on Machine Learning Research (07/2023)
the following presentation applies to all layers and feature channels which BN operates. At training time, BN first computes the mean and the standard deviation across the batch and spatial dimensions, i. e.,
o²
(5)
BHW
BHW
The normalized features è follow from applying these statistics:
(Ii.j.k 11)
(6)
V
Notably, this process differs from the normalization used at inference time. At training time, every BN layer maintains running estimate of μ and across the training batches, which we denote here as f and &. At test time, it is an established practice to normalize the feature values with f and in Eq. 6 instead of the test-batch statistics. We refer to this scheme as train BN (t-BN).
BN and generalization. In the context of out-of-distribution generalization, the running statistics á and ē derive from the source data and can be substantially different had they been computed using the target images. This discrepancy is generally known as the covoriate shift problem. Domain adaptation methods, which assume access to the (unlabelled) target distribution, often alleviate this issue with technique referrod to as Adaptive Batch Normalization (AdaBN; Li et al. 2017b). The key idea behind the method is to simply replace the source rumning statistics with those of the target. Instead of alternating BN layers between the training and testing modes, recent work (Nado et al. 2020) studics prediction- replaces the running statistics from the training time, ft and ô, with the statistics μ and a of the current test batch (normally comprising more than one image). Such a seemingly innocuous change is shown to benefit. model robustness for image classification (Nado et al. 2020 Schneider et al. 2020
BN (p-BN), which
In contrast to AdaBN and p-BN, which utilize either the whole target distribution or a number of samples, our study of model generalization assumes only single target example to be available the one that our model receives as the input at inference time. A viable alternative to AdaBN and p-BN is to compute the statistics per sample, which amounts to replacing BN layers with Instance Normalization (IN) layers (Ulyanov et al.) 2016) after model training. However, this may cause another extreme scenario for covariate shift. Firstly, the sample statistics may serve only as an approximation to the complete test distribution. Secondly, such a replacement may significantly interfere with the statistics of the activations in the intermediale layers with which the network was trained, It is, therefore, unsurprising that IN layers alone were found only hurt the discriminative power of the model in previous work (Pan et al. 2018
Self-adaptive normalization (SaN). Let the source mean ر., and the variance o² denote the running average of the sample statistics at training time. If we had the target domain knowledge expressed by the sufficient statistics ia and 6%, we could use those in place of A. and o in the BN layers to compensate for the covariate shift. However, at test time we only have access to the sample estimates, μt and of provided by a single datum from the target distribution:
Me =
Σ 23 3,k
of
Σ (z m1)² 3.k
HW
HW
where 2 E H.W. is a spatial feature channel in a CNN of the target sample. We define E [0, 1], which denotes the change from the source (a 0) to reference (a 1) image domain. At inference time we compute the new mean and variance, fèr and â, as follows:
(1= + ajt,
2-- -1- to?.
(8)
Using Eq. 6), we replace y and 0² with the computed ia and 67 to normalize the features of the single test sample. Notably, this does not affect the behavior of the BN layers at training time and applies only at test time. Since this approach combines the inductive bias coming in the form of the rumning statistics from the source domain with statisties extracted from a single test instance, which is an unsupervised process, we refer to it as Self-adoplive Normalization (SaN).
5

---

Published in Transactions on Machine Learning Research (07/2023)
In Sec. 5.1 empirically verify that SaN consistently boosts the segmentation accuracy in the out-of- distribution ion scenario. Furthermore, we also find significant improvements of model calibration in terms of the expected calibration error (ECE; Naeini et al., 2015
Related methods. Setting a = 0 in Eq. S defaults to the established procedure, t-BN, and uses only the running statistics from training on the source domain at inference time. Conversely, a = corresponds to Instance Normalization (Ulyanov et al.) 2016) or, equivalently, to the p-BN strategy (Nado al. 2020) with batch size of 1. While je and o, are averaged over a batch of target images in (Nado et al.) 2020 we rely a single test sample in this work. Our experiments in Sec. 5.1 expand upon previous analyses (Schneider ct al. 2020) with an extensive emperical study of this normalization strategy for semantic segmentation. Batch Instance Normalization (Nam & Kim 2018) used a similar weighting approach for adaptive sty lization.. However, a was a training parameter, whereas we use a only for model selection using a validation set.
4 Designing a principled evaluation
Previous studies (Chen et al. 2020 2021: Pan et al., 2018: Yue et al. 2019) on domain generalization for semantic segmentation used divergent evaluation methodologics, which exacerbates the comparison and reproducibility in follow-up research. Encouraged by similar observations (Gulrajani & Lopez-Paz, 2021), we first set out and then follow a number of principles reflecting the best-practice experimentation to enable a fair and reproducible evaluation.
Pan et al. 2018) used only a single target domain, Cityscapes (Cordts et al. 2016), for testing. However, like other rescarch datasets, Cityscapes (Cordts ct al.] 2016) is carcfully curated (e. g., the same camera hardware and country was used for image capture), hence only partially represents the visual diversity of the world. A more comprehensive approach by (Yue et al. 2019) considered a number of target domains. However, separate model was selected for every target domain (based on a different validation set). As a result, each selected model may be biased toward the chosen target domain, hence may not be indicative of strong generalization. This leads us to the next principle:
(A) The lest sel must comprise mulliple domains. A few previous works (Chen el al. 2020 2021
(B) A single model must be used for all test domains. This principle follows naturally from the definition of generalization, i. e. the ability of the model to reliably operate (e. g. in terms of accuracy) under varying test environments. To produce a -pmain-specific model is the goal of another research avenue, domain adaptation.
Many works do not clearly define the strategy of model selection, such the used validation set Indeed, the use of test domains for hyperparameter tuning has not been uncommon (e. g. (Huang et al. 2021 This is especially undesirable when studying model generalization. Therefore, we stress that:
(C) Model selection must not use test images; and (D) The validation set must be clearly specified. - These principles follow naturally from the requirements of domain generalization, with principles C and D, in particular, being widely accepted in machine learning research. To our surprise, we found that no previous work on domain generalization for semantic segmentation has yet fulfilled all of these principles. To encourage and facilitate good evaluation practice, we therefore revise the experimental protocol used so far. We consider a practical scenario in which supplier prepares a model for a consumer without a-priori knowledge on where this model may be deployed, much akin to Kundu et al. (2021). On the supplier's side, we assume access to two data distributions for model training and validation, the source data and the validation set. After training the model on the source data and choosing its hyperparameters on the validation set, we assess its generalization ability on three qualitatively distinct target sets. The average accuracy across these sets provides an estimate of the expected model accuracy for its out-of-distribution deployment on the consumer's 's side. Next, we concretize the datasets used in this study, which focus traffic scenes for compatibility with previous work (Chen et al. 2021 Pan et al. 2018 Yue et al.] 2019). Notably, the scale of our study, summarised in Table exceeds that of previous work.
In contrast, our evaluation methodology adheres to the principles of robustness and generalization by testing a single model on multiple target domains using a single set of hyperparameters without access to images from respective target datasets. This ensures that our model can perform well across range of domains without, the need for domain-specifie fine-tuning or model selection.
6

---

Published in Transactions on Machine Learning Research (07/2023)
Table State- Saade-of-ttee the--trr domain generalization methods for reported source domains and target domains.
Source Domains
Target Domains
Method
GTA SYNTHIA
CS
Mapillary
BDD
IDD
DRPC (Yue et al.
2019
ASG CCen et al.
2020
CSG Chen al.
2021
Robust.eet (Choi al.
2021
FSDR (Huang a..
2021
WildNet (Lee et a.l
2022b
SAN-SAW (Peng et al.
2022
PIN (Kim et al. 2022
GCISG Nam et al.
2022)
XDED Lee et al.)
2022a
This work
5
Experiments
Following our discussion above, we revise the evaluation protocol as follows. On the supplier's side, we assume access to two data distributions for model training and validation, the source data and the validation set. We assoss the generalization ability of the model yiclded by the validation process on three qualitatively distinct. largel sels. The average accuracy across these sets provides an estimate of the expected model accuracy for its out-of-distribution deployment on the consumer's side. Next, we concretize the datasets used in this study,
which are restricted to traffic scenes for compatibility with previous work (Chen et al. 2021 Pan et al. 2018 Yue et al. 2019) (sce supplemental material for dataset details).
Source data. We train our model on the training split of two synthetic datasets (mutually exclusive) with low-cost ground truth aunotation: GTA (Richter et al.) 20166) and SYNTHIA (Ros et al. 2016). Importantly, these datasets exhibit visual discrepancy (i. e., domain shift) w. r. the real imagery, which our model needs to generalize.
Validation set. For model selection and hyperparameter tuning, use the validation set of WildDash (Zndel et al. 2018). In our scenario, the validation set is understood to be of limited quantity, owing to its more costly annotation compared to the source data. In contrast to the training set, however, it bears closer visual resemblance to the potential target domains.
Multi-target evaluation. Following model selection, we evaluate the single model on three target domains comprising the validation sets from Cityscapes (Cordts et al. 22016), BDD (Yu al. 2020),, and IDD (Varma) et al. 2019 The choice of these test domains stems from mumber of considerations, such the geographic origin of the scenes (Cityscapes, BDD, and IDD were collected in Germany, North America, and India, respectively). Geographic distinction as well as substantial differences in data acquisition (e. g., camera properties) of these datasets bring together an assortment of challenges for the segmentation model at test time. Since the deployment site of our model is unknown, we assume a uniform prior over the target domains as our test distribution. Under this assumption, the average of the mean accuracy across our target domains estimates the expected model accuracy.
To compare to previous works, also evaluate Mapillary (Neuhold et al. 2017 Mapillary does not publicly disclose the geographic origins of individual samples, hence is unsuitable to identify a potential location bias acquired by the model from the training data. This is possible in our proposed evaluation protocol, since the geographic locations from Cityscapes, BDD, and IDD do not overlap.
Implementation details. We implement our framework PyTorch (Paszke et al., 2019). Our code and pre-trained models are publicly available. We also discuss trivial to implement, but crucially useful training details of our baseline in-depth here and in Appendix A

---

Method Published in Transactions on Machine Learning Research (07/2023)
Table 2: (a) Segmentation accuracy using SaN. We report the mean IoU (%, 1) on three target domains
(Cityscapes, BDD, IDD) across both backbones. t-BN denotes train BN (Ioffe & Szegedy 2015). while
p-BN refers to prediction-time BN (Nado al. 2020). (b) ECE (%, 4 for SaN and MC- (Gal &
Ghahramani 2016). We trained the networks on GTA and report scores for the three target domains (ef.
Appendix B.1 for results with SYNTHIA).
(a)
ToU (%, t)
(b)
ECE
(%, +)
Method
CS BDD
IDD
Mean
CS
BDD
IDD
Mean
ResNet-50
ResNet-50
37.28
35.61
27.73
33.54
w/ 1-BN
30.95
28.52
32.78
30.75
w/ SaN (Ours)
30.57
30.94
26.90
29.47
W/ p-BN
37.71
31.67
30.85
33.41
w/ MC-Dropout
30.29
29.80
24.17
28.09
w/ SaN (Ours)
37.54
32.79
34.21
34.85
w/ both (Ours)
25.50
27.36
22.62
25.16
ResNet-101
ResNet-101
35.24
33.74
27.28
32.09
w/ t-BN
32.90
32.54
30.36
31.93
w/ SaN (Ours)
26.12
28.89
23.98
26.36
W/ p-BN
39.88
34.30
33.05
35.74
w/ MC-Dropout
31.30
29.95
25.15
28.80
w/ SaN (Ours)
42.17
35.40
33.52
37.03
w/ both (Ours)
24.44
28.68
23.32
25.48
Following (Pan et al. 2018), our baseline model is DeepLabvl (Chen et al. 2015 without CRF post-processing, but the reported results also generalize to more advanced architectures (ef. Appendix B.3 We use ResNet-50 and ResNet-101 (He et al. 2016) pre-trained on ImageNet (Deng et al., 2009) as backbone. We minimize the cross-entropy loss with an SGD optimizer and a learning rate of 0.005, decayed polynomially with the power set to 0.9. All models are trained on the source domains for 50 epochs with batch size, momentumm, and weight decay set to 4, 0.9, and 0.0001, respectively. For data augmentation, we compute crops of random size (0.08 to 1.0) of the original image size, apply random aspect ratio (3/4 to 4/3) to the crop, and resize the result to 512 512 pixels. We also use random horizontal flipping, color jitter, random blur, and grayscaling. We train our models with SyneBN (Paszke et al. 2019 on two NVIDIA GeForce RTX 2080 GPUs.
5.1 SaN improves segmentation accuracy and prediction uncertainty
For both source domains (GTA, SYNTHIA) in combination with all main target domains (Cityscapes, BDD, IDD), we investigate the influence of a on Self-adaptive Normalization (SaN, Eq. g in terms of the IoU. Setting an optimal a for every larget domain is infeasible in domain generalization as the target domain during inference is unknown. Instead, we choose the optimal a in steps of 0.1 based on the IoU on the validation set of WildDash. For the ResNet-50 backbone, we attain the highest validation IoU for both training datasets with a = 0.1 (see Appendix B.Ip. Fixing this optimal a, we proceed with evaluating our model on the target domains. Table reports the segmentation accuracy with this a that has been optimized on the validation set. In Table 2 GTA Cityscapes, BDD, and IDD and compare the accuracy of the larget domains with (-BN and p-BN. Remarkably, SaN improves the mean IoU not only of the t-BN baseline (c. g., by 4.1% loU with Net-5t))50) which represents an established evaluation mode, but also over the more recent p-BN (Nado et al. 2020 This improvement is consistent across the board, i. e. irrespective of the backbone architecture and the target domain tested. Furthermore, we found that the calibration of our models, in terms of the expected calibration error (ECE; Naeini et al., 2015), also improves. As shown in Table 2(b,, not only does SaN substantially enhance the bascline, but even competitive with the commonly used MC-Dropout method (Gal & Ghahramani] 2016). Rather surprisingly, SaN exhibits a complementary effect with MC-Dropout: the calibration of the predictions improves even further when both methods are used jointly.
(a) we report loU scores for both backbones on generalization from
5.2 Self Self-adaptation vs. Dist-Time Augmentation (TTA)
We compare our solf-adaptation to the standard non-adaptive inference, as well as test our models against Test Time Augmentation (TTA; Simonyan & Zisserman 2015) as a stronger baseline. TTA augments the test samples with their flipped and grayscaled version multiple scales and averages the predictions as the
8

---

Published in Transactions on Machine Learning Research (07/2023)
Table 3: Meon IoU (%) with TTA (Simonyan & Zisserman 2015) and our self-adaptation reported across both source domains (GTA, SYNTHIA) and three target domains (Cityscapes, BDD, IDD).
Source:
GTA
Source: SYNTHIA
Method
CS
BDD
IDD
Mean
CS
BDD
IDD
Mean
ResNet-50 (w/ SaN)
37.54
32.79
34.21
34.85
36. 14
26.66
26.37
29.72
TTA (w/ SaN)
42.56
37.72
37.98
39.42
39.67
32.10
30.46
34.08
Selfaddapaation (Ours)
45.13
39.61
40.32
41.69
41.60
33.35
31.22
35.39
ResNet-101 (w/ SaN)
42.17
35.40
33.52
37.03
38.01
28.66
27.28
31.32
TTA (w/ SaN)
44.37
38.49
38.35
40.40
39.91
32.68
30.04
34.21
Self-adaptation (Ours)
46.99
40.21
40.56
42.59
42.32
33.27
31.40
35.66
final result. For self-adaptation, we use horizontal flipping and grayscaling with factor scales of (0.25, 0.5, 0.75) w. r. t. the original image resolution. We study the relative importance of these augmentation types in Appendix B.22 Based on the validation set WildDash, we set threshold 1 = 0.7, N₂ = 10 iterations, and a learning rate 7 0.05. We only train the layers conv4 x, conv5 x, and the classification head as we did not observe any benefits from updating all model parameters. Furthermore, this reduces runtime due to not backpropagating through the whole network. We investigate this choice part of the nutime-accurace trade-off in Sec. 5.4 In Table 3 we show IoU scores for both souree domains (GTA, SYNTHIA) and three larget domains (Cilyscapes, BDD, IDD) across both backbones. Even though TTA improves the baseline (e. g., by 3.37% ToU with ResNet-101 using GTA), our proposed self-adaptation still outperforms it by a clear and consistent margin 2.19% loU on average. This observation aligns well with our reported ECE scores in Table b)) demonstrate that self-adaptation further exploits the calibrated confidence of our predictions yield reliable pseudo-labels for adapting the model to particular test sample.
5.3 Comparison to state of the art
We compare self-adaptation with trate-of-tho-rr domain generalization methods in Table + While most of the other methods report their results on weakly tuned baselines, we show consistent improvements even over a carefully tuned bascline, regardless of backbone architecture source data. Our single model with self-adaptation even outperforms DRPC Yue et al. (2019) and FSDR Huang et al. (2021) on most benchmarks (e. g., by 4.2 9.4% on Mapillary with ResNet-
These methods train individual models for each target domain; FSDR Huang et al. (0021) resorts to the target domains for hyperparameter tuning, hence contravenes our proposed
evaluation protocol. Recall that ASG Chen et al. (2020) and CSG Chen el al. (2021) (as well as DRPC; Yue el al. 2019) require access to a dist.riuttinn real images for training, while NBN-Net Pan et al. (2018) modifies the model architecture. Our thoroughly straighttorward approach requires neither, alters only the inference procodure, yet outperforms these methods in all benchmark scenarios substantially. WildNet Lee et al. (2022b) appears to be more accurate than self-adaptation on BDD with Re1 However, this is not a fair comparison, since it uses a more advanced architecture (DeepLabv3+ us. DeepLabvl). As we will see in Table 12] self-adaptation, in fact, outperforms WildNet when using the same architecture by 2.79% IoU. Similarly, SAN-SAW [eng et al.] (2022) reaches higher accuracy on BDD, trained on SYNTHIA, presumably due to the ASPP module Chen et al. (2018a) that we do not use. Self-adaptation considcrably outperforms SAN-SAW in all other scenarios. Overall, despite adhoring to stricter evaluation practice and simpler model architecture, self-adaptation overwhelmingly exceeds the segmentation accuracy of previous work.
Comparison to Tent (Wnng et al., 2021a). Like self-adaptation, Tent (Wng et al. 2021a) also updates model parameters al lest time. However, different from constructing the pseudo-labels based on well-calibrated predictions in our self-adaptation, Tent simply minimizes the entropy of a single-scale prediction. Tent also limits the adaptation to updating only the BN parameters, whercas our self-adaptation generalizes this process to convolutional layers. To demonstrate these advantages, we compare to Tent (Wang et al. [0021a] in Table 5 We train HRNet-W118 (Wang et al. 2021bb on GTA and compare the loU on Cityscapes to the
9

---

Published in Transactions on Machine Learning Research (07/2023)
Table 4: Mean IoU (%, t) comparison to state-of-the-ard domain generulization methods for both source domains (GTA, SYNTHIA) as well three target domains (Cityscapes, Mapillary, BDD). In-domain training obtain the upper bounds uses our baseline DeepLabyl following the same schedule as with the synthetic datasets. (f), (t), (#) and (tt) denote the use of FCN (Long et al. 2015), DeepLabv2 (Chen et al. 201 8a).
DeepLabv3 Chen et al. 2017). and D (Chen et al. 2018b), respectively.
Backbone:
Re
Backbone: ResNet- 101
Method
CS
Mapillary
BDD
CS
Mapillary
BDD
In-domain Bound
71.23
58.39
58.53
73.84
62.81
61.19
No Adapt
32.45
25.66
26.73
33.56
28.33
27.76
14.97
18.46
15.41
18.97
19.72
10.96
DRPC* (Yuc ct al.
2019
37
34.12
32.14
42.53
38.05
38.72
No Adapt
29.63 63
13.77
18.16
ASG! [Chen et
2020
29.65
32.79
No Adapt
25.88
29.63
19.39
19.25 38.88
CSG (Chen
2021
35.27
No Adapt
28.95
28.18
25.14
RobustNet!! CCoo et al.
2021
36.58
17.63
40.33
112.15
35.20
110.06
GTA
No Adapt
33.4
27.9
27.3
111.4
115.5
113.9
FSDR* (Huang et al.
2021
44.8
43.4
41.20
No Adapt
35.16
19.46
31.29
114.77
29.71
118
35.73
110.06
33.42
113.66
34.06
17.67
WildNettt
Lec et al.
2022b
44.62
46.09
38.42
45.79
47.08
41.73
No Adapt.
29.32
28.33
25.71
30.64
28.65
27.82
SAN-SAW Pong al.
2022
39.75
110.49
41.86
113.53
37.34
111.65
45.33
114.69
40.77
122.12
41.18
119.96
No Adapt
PIN (Kim et al. 2022
31.60
41.00'1
19.40
19.40
29.00
37.40f1
18.40
34.60tt1..50
25.10
34.60f1
19.50
44.901
39.711
41.31
No Adapt
26.67
112.34
GCISC
Nam et al.
2022
39.01
No Adapt
28.90
28.10
25.10
10.30
19.00
17.30
XDED
Lee et al.
2022a
39.20
37.10
32.40
No Adapt
30.95
114.18
34.56
112.93
28.52
11..09
32.90
144.09
36.00
11.49
32.54
17.67
Self-adaptation (Ours)
45.13
47.49
39.61
46.99
47.49
40.21
No Adapt.
28.36
27.24
25.16
29.67
28.73
25.64
17.29
15.50
16.37
17.91
5..39
18.70
DRPC (Yue et al. 2019
35.65
32.74
31.53
37.58
34.12
34.34
SYNTHIA
No Adapt
FSDRE (Huang al.
2021
40.8
39.6
37.4
No Adapt
23.18
21.79
24.50
23.
21.84
25.01
SAN-SAW (Peng et al.
2022
38.92
115.74
34.52
1
35.24
110.74
40.87
17.02
37.26
15.42
35.98
10.97
Self-adaptation (Ours)
No Adapt
31.83
41.60
19.77
41.21
33.41
17.80
33.35
24.30
19.05
42.32
37.25
15.07
36,84
41.20
14.36
33.27
29.32
13.95
Table 5: Mean ToU (%, +) comparison to Tend [Wang el al. 2021ah. (a) Using the same HRNet-W18 backbone (Wang et al. 2021bb in (Wang et al. 2021a),, our self-adaptation outperforms Tent Wang
et al. 2021a substantially, even when using SaN alone. (b) Our self-adaptation loss defined in Sec. 3 yields
significantly higher segmentation accuracy compared to the entropy minimization used by Tent Waag et al. 2021a).
HRNet-WWi W18 backbone
(b) Comparison to entropy minimization (DeepTabvl)
Mothod
CS
Method
CS
BDD
IDD
Mean
Tent (Wagg et al. 2021a
36.4
Entropy min.
40.20
35.40
33.97
36.52
SaN (Ours)
40.0
Ours (v==0.0)
44.00
38.55
39.09
40.55
Self-adaptation (Ours)
44.1
Ours (0=00.7)
45.13
39.61
40.32
41.69
equivalent configuration of Tent. Under a comparable computational budget of model update itcrations, we substantially outperform Tent, by a remarkable 7.5% IoU. SaN alone already outperforms Tent significantly with single forward pass by 3.6%. Further, our self-adaptation loss is also considerably more effective
10

---

Published in Transactions on Machine Learning Research (07 (07/2023)
45
TTA
TTA
loU (%)
40
SSaN
SaN
Ours (conv4_
Ours conv5 Head)
35-
Ours (Ieead)
Baseline
30
Baseline
Model Ensemble
3
8
11
Runtime (s)
Remlisee-ccuuracy comporison on GTA + Cilyscppes generalization using one NVIDIA GeForce RTX 2080 GPU. The curves trace self-adaptation iterations, e., the first point corresponds to N₁ while the last shows N, 10. Self-adaptation balances accuracy and inference time by adjusting iteration numbers and layer choices, and is more cost-effective than 10 network ensembles.
Figure 2:
0.05, N2 = 10
0.7, N, 10
45.0
loU (%)
37.5
45.0
42.5
32.5
Sa.
40.0
Sell-Adaplation
44.5-
Adlll-Aaptttionn
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.02
0.04
0.06
0.08
0.10
Weight
Threshold
Learning Rate
Figure 3: Hyperporameter sensitivity on GTA + Cityscapes generalization. We investigate our hyperparam- etcrs a, threshold & and lcarning rate yj and report scores using Docplabyl with the ResNet-50 backbone trained on GTA. For self-adaptation, we one hyperparameter (v, v) while varying the other.
than the entropy minimization employed by Tent. We improve by 4% IoU on average even without tuning threshold v. and by 5.2% when it is tuned.
5.4 Additional analysis
Runtine-accurac, trade-off. We investigate the influence of the number of iterations required to adapt to single sample during self-adaptation. Fig, 2|etts IoU scores for Cityscapes using the ResNet-50 backbone trained on GTA (Appendix B.2 provides a numerical comparison). As widely adopted baseline, we also train a model ensemble comprising 10 networks with a DeepLabv1 architecture (as in self-adaptation). initialized with a random seed (Hansen & Salamon! 1990). Note that TTA, selfaadaptation, and the ensemble use SaN for fair comparison, and we also test the ensemble with TTA. Although self-adappation increases the accuracy of the baselines at the expense of test-time latency, is still more efficient and more accurate than trade off the accuracy runtime by
the model ensembles. Another advantage is that self-adaptation using fewer update iterations, or updating fewer upper network layers. While the top-accuracy variant of self-adaptation may not be suitable for real-time applications yet, it can still be valuable in other important domains, such as medical imaging, where high accuracy is desirable even at the cost increased latency. For real-time needs, SaN alone boosts the baseline accuracy significantly without any computational overhead.
Hyperparameter sensitivity Fig. plots segmentation accuracy w. r. t. our hyperparameters a, v and m. Complying with protocol in Sec. 1 exclusively rely follows. As detailed in Sec.
validation set to find the optimal values first use SaN alone and determine leading to the highest accuracy the validation set, yiclding a = 0.1. Recall that the threshold & plays a crucial rolc in determining the reliability of our pseudo-labels, while the learning rate governs the test-Lime adaptation process. We use grid search and fine- tune network parameters with adlf-adaptttion on the validation set, which establishes the optimal values 1 = 0.7 and 7 = 0.05. We observe from Fig. 3 that the model accuracy on the validation set declines moderately we deviate from the optimal hyperparameter values. For example, we were increase n twofold, the accuracy would drop by only 0.15% ToU. Appendix B.2 provides numerical results to reproduce
5.1 [.1]
11

---

Published in Transactions on Machine Learning Research (07 (07)))
GTA +
Cityscapes
GTA -
BDD
GTA + Mapillary
GTA 7 IDD
Imagee
Ground Truth
Baseline
Ours
Figure 4: Qualitative semantic segmentation results for the generalization from GTA to Cityscapes, BDD, Mapillory, and IDD for the ResNet-50 backbone. We show the input image (lop row), ground truth and the predictions of the baseline model and of our proposed self adlptatin (bottom row).
Fig. 3 Overall, hyperparameters in self-adaptation exhibit a reasonable range of tolerance. Our comparision to the state of the art in Sec. 5.3 also supports this conclusion, since these hyperparameter values may not be optimal for the larget domains. Self-adaptation achieves superior segmentation accuracy nonetheless.
Qualitative results. In Fig, we visualize qualitative segmentation results produced by selfaddaptation for generalization from GTA to Cityscapes, BDD, Mapillary, and IDD. We observe clearly perceivable improvement over the baseline, especially in terms of consistency with the image boundaries. Appendix B.5 illustrates further examples and discusses failure cases.
6 Conclusion
The f.idd. assumption underlying the traditional learning principle, ERM, implies that training process relying it is unlikely to produce models robust to an arbitrary domain shift, unless we make further assumptions about the test distribution. In the out-of-distribution scenario, the test domain is unknown, hence formulating such assumptions difficult. To bypass this issue, we presented and studied a self-adaptive inference process. We also highlighted number of shortcomings the experimental protocol used in previous work. By following the best practice machine lcarning research, we formulated four principles defining a rigorous evaluation process in domain generalization. We implemented and followed these principles in our experiments.
Our analysis demonstrates that single sample from the test domain can already suffice to improve model predictions. The accuracy improvement shown by our experiments is surprisingly substantial, despite using a fairly straightforward approach without changes to the training process or the model architecture, unlike in previous works (Chen al. 2020 Yue al. 2019). We hope that these encouraging results will incentivize our research community to study self-adaptive techniques in other application domains, such as panoptic segmentation, monocular depth prediction.
12

---

Published in Transactions on Machine Learning Research (07/2023)
The particular instantiation of self-adaptation that we presented in this work is not yet real-time. We extensively analyzed the existing trade-off with the segmentation accuracy in Sec. 5.4 and Appendix B.2 and found self-adaptation to be nonetheless more cost-effective than model ensembles framework in machine learning. However, the scope of this work was to rigorously demonstrate concrete accuracy benefits, which will make improving model efficiency a worthwhile goal to pursue in future work.
widely adopted
Indeed, decreasing the lateney of self-adaptive inference is an intriguing avenue for research. Using adaptive step sizes, higher-order optimization (Botev et al,, 2017 Kingma & Baj 2015), or implicit gradients (Rajeswaran et al. 2019 we may lower the number of required update steps. An orthogonal line of research may consider low-precision computations (Wang ct al. 2019), low-rank decompasition (Jaderberg et al. 2014], or alternating update strategies [Tonioni et al. 2019 to offer reduced computational footprint for each update iteration. We are excited to explore these directions and hope the reader may find this work equally inspiring.
Acknowledgments
This project is partially funded by the European Research Council (ERC) under the European Union's Horizon 2020 rescarch and innovation programme (grant agrecment No. 866008) as well as the State of Hesse (Germany) through the cluster project "The Adaptive Mind (TAM)".
References
Nikita Araslanov and Stefan Roth. Self-supervised augmentation consistency for adapting semantic segmen- tation. In CVPR, pp. 15384 15394, 2021.
Martín Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Pas Paz. Invariant risk minimization.
[stat.ML]. 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.06433 [LL, 2016.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory learning from different domains. Mach. Learn., 79(1-2):151-175, 2010.
Steffen Bickel, Michael Brückner, and Tobias Scheffer. Discriminative learning under covariate shift. J. Mach. Learn. Res., 00-217722155 2009.
Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical Gauss-Newion optimisation for deep learning. In ICML, volume 70, pp. 557-566 565, 2017.
Minjie Cai, Feng Lu, and Yoichi Sato. Generalizing hand segmentation in egocentric videos with uncertainty- guided model adaptation. In CVPR, pp. 14392-14401, 2020.
Debora Caldarola, Barbara Caputo, and Marco Ciccone. Improving generalization in federated learning by seeking flat minima. In ECCV, pp. 654 672, 2022.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Semantic image segmentation with deep convolutional nets and fully connected CRFs. In ICLR, 2015.
Liang-Chich Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXim: 17Xim:1706.055 706.055677777 (cs. CVJ, 2017.
Liang- Chinh Chen, George Papandreou, Tasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans. Pattern Anol. Mach. Intell,, 40(4):834 848, 2018a.
Slang-Chich Chen, Yukum Zhu, George Papandrcou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, volume 7, pp. 833 881-851, 2018b. 8b..
13

---

Published in Transactions on Machine Learning Research (07/2023)
Wuyang Chen, Zhiding Yu, Zhangyang Wang, and Animashree Anandkumar. Automated synthetic-to-real generalization. In ICML, volume 119, pp. 1746-1756, 2020.
Wuyang Chen, Zhiding Yu, Shalini De Mello, Sifei Liu, Jose M. Alvarez, Zhangyang Wang, and Anima Anandkumar. Contrastive syn-to-real generalization. In ICLR, 2021.
Sungha. Choi, Sanghun Jung, Huiwon Yun, Joanne T Kim, Seungryong Kim, and Jaegul Choo. NebustNel: Improving domain generalization in urban-scene segmentation via instance selective whitening. In CVPR, pp. 11580 11590, 2021.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Scharwächter, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schicle. The Cityscapes dataset for semantic urban scene understanding, In CVPR, pp. 3213 3223, 2016.
Lucas Deecke, lain Murray, and Hakan Bilen. Mode normalization. In ICLR, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. ImageNet: A large-scale hierarchical image database. CVPR, pp. 248-255, 2009.
Chelsea Finn, Picter Abbeel, and Sergey Levine. Model-agnostic meta-lcarning for fast adaptation of deep networks. In ICML, volume 70, pp. 1126 1135, 2017.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In ICML, volume 48, pp. 1050~1059, 2016.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-advvesarrial training of neural networks. J. Mach. Learn. Res., 17:59:1-59:355 2016.
Daniel Glasner, Shai Bagon, and Michal Irani. Super-resolution from single image. In ICCV, pp. 349-356, 2009.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In ICLR, 2021.
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE Trans. Pottern Anal. Mach. Intell., 12(10):99331000 1001, 1990.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. FSDR: Frequency space domain randomization for domain generalization. In CVPR, pp. 6891 1991-990,,40 2021.
Lei Huang, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Iterative normalization: Beyond standardization towards efficient whitening. In CVPR, pp. 4874-4883, 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, volume 37, pp. 448 4484456,44 2015.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. In BMVC, 2014.
Zdenek Kalal, Krystian Mikolajezyk, and Jiri Matas. Mach. Intell., (09))):44911422 2012.
IEEE Thans. Pattern Anal. Jin Kim, Jiyoung Lee, Jungin Park, Dongbo Min, and Kwanghoon Sohn. Pin the memory: Learning to generalize semantic segmentation. In CVPR, pp.
2022.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Jogendra Nath Kundu, Akshay Kulkarni, Amit Singh, Varun Jampani, and Venkatesh Babu. Generalize then adapt: Source-free domain adaptive semantic segmentation. In ICCV, pp. 7046-7055 2021.
14

---

Published in Transactions on Machine Learning Research (07/2023)
Kyungmoon Lee, Sungyeon Kim, and Suha Kwak. Cross-domain ensemble distillation for domain generalization. In ECCV, pp. 1-20, 2022a.
Subyeon Lee, Hongje Seong, Seongwon Lee, and Euntai Kim. WildNet: Learning domain generalized semantic segmentation from the wild. In CVPR, pp. 9936 9946, 2022b.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In ICCV, pp. 3542-5550, 2017a.
Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. In JCLR, 2017b.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pp. 9992 10002, 2021.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, pp. 3431-3440, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan. Unsupervised domain adaptation with residual transfer networks. In NeurIPS, pp. 136-144, 2016.
Ping Luo, Jiamin Ren, Zhanglin Peng, Ruimao Zhang, and Jingyu Li. Differentiable learning-to-normaliza via switchable normalization. In ICLR, 2019.
Grégoire Montavon, Genevieve B. Orr, and Klaus-Robert Müller (eds.). Neural Networks: Tricks of the Trade Second Edition, volume 7700 of Lecture Notes in Computer Science. Springer, 2012.
Zachary Nado, Shreyas Padhy, D. Sculley, Alexander D' D'Amour, Balaji Lakshminarayanan, and Jasper Snock.
Evaluating prediction- batch normalization for robustness under covariate shift. 1rX6::2006.10963 Les. 2020.
Mahdi Pakdaman Nacini, Gregory F. Cooper, and Milos Hauskrocht. Obtaining well calibrated probabilities using Bayesian binning, In AAAI, pp. 2901 2907, 2015.
Gilhyun Nam, Gyeongjae Choi, and Kyungmin Lee. Gcisg: Guided causal invariant learning for improved syn-to-real generalization. In ECCV, pp. 656 672, 2022.
Hyeonseob Nam and Hyo-Eun Kim. Batch-instance normalization for adaptively style-invariant neural networks. In NewrIPS, pp. 2563-2572 2018.
Gerhard Ncuhold, Tobias Ollmann, Samuel Rota Buld, and Peter Kontschicder. The Mapillary Vistas datasct for semantic understanding of street scenes. ICCV, pp. 4990 4999, 2017.
Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two once: Enhancing learning and generalization capacities via IBN-Net. In ECCV, volume 4, pp. 484-500 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. In NewrIPS, pp. 8024 8035, 2019.
Duo Peng, Yinjie Lei, Munawar Hayat, Yulan Guo, and Wen Li. Semantic-aware domain generalized segmentation. In CVPR, pp.
2022.
Aravind Rajeswaran, Chelsea Finn, Sham M. Kakade, and Sergey Levine. Meta-learning with implicit gradients. In NewrIPS, pp. 113 124, 2019.
Nikhil Reddy, Abhinav Singhal, Abhishck Kumar, Mahsa Baktashmotlagh, and Chetan Arora. Master of all: Simultancous generalization of urban-scene segmentation to all adverse weather conditions. In ECCV, pp. 51-69, 2022.
15

---

Published in Transactions on Machine Learning Research (07/2023)
Stephan R. Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In ECCV, volume 2, pp. 102-118, 2016.
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M. Lopez. The SYNTHIA dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR, pp- 3234 3243, 2016.
Christos Sakaridis, Dengxin Dai, and Lue Van Gool. Acde: The adverse conditions dataset with correspon- dences for semantic driving scene understanding. In CVPR, pp. 10765 10775, 2021.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wicland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. In NewrIPS, pp. 11539 11551, 2020.
Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. SinGAN: Learning a generative model from a single natural image. In ICCV, pp.
2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efrrs, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In ICML, volume 119, pp. 9229 9248, 2020,
Zhiqiang Tang, Yunhe Gao, Yi Zhu, Zhi Zhang, Mu Li, and Dimitris N. Metaxas. CrossNorm and SelfNorm for generalization under distribution shifts. In ICCV, pp. 52 61, 2021.
Scbastian Thrun. Lifelong learning algorithms. In Learning to Learn, pp. 181-209. Springer, 1998.
Alessio Tonioni, Fabio Tosi, Matteo Poggi, Stefano Mattoccia, and Luigi di Stefano. Real-time self-adaptive deep stereo. In CVPR, pp. 195-204, 2019.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR, pp. 1521 1528, 2011.
Dmitry Ulyanvv, Andrea Vedaldi, and Victor S. Lempitsky. Instance normalization: The missing ingredient. for fast stylization. arXiv:
(cs.CV], 2016.
Girish Varma, Anbumani Subramanian, Anoop M. Namboodiri, Manmohan Chandraker, and C. V. Jawahar. IDD: A dataset for exploring problems of autonomous navigation in unconstrained environments. In WACV, pp. 1751-1751, 2019.
Thomas Varsavsky, Mauricio Orbes-Arteaga, Carole H Sudre, Mark S Graham, Parashkev Nachev, and M Jorge Cardoso. Test-time unsupervised domain adaptation. In MICCAI, pp. 428-444--00 088-436,550 2020.
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C. Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In NeurIPS, pp. 5339-53333 2018.
Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In CVPR, pp. 2517-2526, 2019.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021a.
Guangrun Wang, Jiefeng Peng, Ping Luo, Xinjiang Wang, and Liang Lin. Kalman normalization: Normalizing internal representations across network layers. In NewrIPS, pp. 21-31, 2018.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borni Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation learning for visual recognition. IEEE Trans. Pallern Anal. Mach. Inlell, (3310))3349-3300 2021b.
Yue Wang, Ziyu Jiang, Xiaohan Chen, Pengfei Xu, Yang Zhao, Yingyan Lin, and Zhangyang Wang. E2-train: Training state-of-the-art CNNs with over 80% energy savings. In NewrIPS, pp. 5139-516 019-5151, 2019.
16

---

Published in Transactions on Machine Learning Research (07/2023)
Gerhard Widmer and Miroslav Kubat. Learning in the presence of concept drift and hidden contexts. Mach. Learn., 23(1))69-101, 1996.
Yuxin Wu and Kaiming He. Group normalization. In ECCV, volume 8, pp. 3-19, 2018.
Tele Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceplual parsing for scene understanding. In ECCV, volume 5, pp. 418-434, 2018.
Shaoan Xie, Zibin Zheng, Liang Chen, and Chuan Chen. Learning semantic representations for unsupervised domain adaptation. In TCML, pp. 5423-5432, 2018.
Yanchao Yang and Stefano Soatto. FDA: Fourier domain adaptation for semantic segmentation. In CVPR, pp. 4084 4094, 2020.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. BDD100K: A diverse driving dataset for heterogeneous multitask learning. CVPR, pp- 2633 2642, 2020.
Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto L. Saugiovammi-Vnccenellli Kurt Keutzer, and Boqing Gong. Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data. In ICCV, pp. 2100 2110, 2019.
Oliver Zendel, Katrin Honauer, Markus Murschitz, Daniel Steininger, and Gustavo Fernández Domínguez. WildDash - Creating hazard-aware benchmarks. In ECCV, volume 6, pp. 407-421, 2018.
Marvin Zhang, Sergey Levine, and Chelsea Finn. MEMO: Test time robustness via adaptation and augmen- tation. In NewrIPS, 2022.
17

---

Published in Transactions on Machine Learning Research (07 2023))
A Baseline: Best practice implementation
To legitimately study the out-of-distribution model accuracy, it is essential to establish its upper bound attainable by the standard training procedures (Montavon et al. 2012). We followed the best practice in the literature and found a number of training details to be crucial for obtaining a highly competitive baseline, i. e., a model that does not use our self-adaptive inference. Among them is using heavy data augmentation. Recall from Soc. that we used random horizontal flipping, multi-scale cropping with scale range of [0.08, 1.0), as well as photometric image perturbations color jitter, random blur, and grayscaling. Color jitter, applied with probability 0.5, perturbs image brightness, contrast, and saturation using factor sampled uniformly from the range 1.3). We use different range of [0.9, 1.1] for the hue factor. We randomly blur the image using a Gaussian kernel with the standard deviation sampled from (0.1,2.0) Additionally, we convert the image to grayscale with a probability of 0.1. Furthermore, we also found that the polynomial decay schedule we used for the learning rate, as well training for at least 50 epochs (for both GTA and SYNTHIA) are essential to achieve a high baseline accuracy. Note that we only used WildDash as the development set to tune these training details. We also experimented with higher imput resolution and a larger batch size, but did not observe significant improvement, yet a drastic increase in the computational overhead.
On importance of the baseline. We note that the reported accuracy from previous work in Table 4 is not entirely consistent w. r. t. the choice of the model architecture, In particular, DRPC (Yue et al. 2019) uses an FCN, yct outperforms other domain genoralization approaches with DeepLaby1 (Pan et al. 2018) and
DeepLabv2 (Chen et al. 2020 2021) considerably, as shown in Table 4 This is a regrettable consequence of inconsistent training schedules used in previous works that proved difficult to reproduce. For example, at the time of submission, the implementation by Yue et al. (uue et al. 2019), which reports excellent segmentation accuracy for the baseline (ef. Table 4 has not yet been made available parts of the code implementing the semantic segmentation architecture introduced in (Pan et al. 2018 are also not publicly available These circumstances make reporting the accuracy of the implenentation-specife basclines indispensable, which
has thus become the standard practice in more recent previous (Chen el al. 2020: 2021) and related works
Guuraaaaii & Lopez-Par Paz 2021).
B Self-adaptation: Additional analysis
B.1 Analyzing self-adaptive normalization (SaN)
Selecting a. Fig. shows a detailed plot of the influence of a on the segmentation accuracy, both on the development set of WildDash and on the larget domains. We observe that the maximum accuracy on the development set is attained with a = 0.1. Clearly, there is no guarantee that value 0.1 is the optimal one for the target domains. However, choosing a based on the development set is in line with the established practice in machine learning: Tuning model hyperparameters is not allowed on the test sets ei, Cityscapes, BDD, IDD), but is only possible on the validation set (WildDash). In general, the hyperparameters found to be optimal on the validation set are not guaranteed to remain so on the test set, especially in the -ut-of-distribution scenario studied here. Nevertheless, self-adaptation is quite robust even to the incvitably suboptimal choice of the hyperparameters in the out-of-distributtin selting. Despite a having been picked based on the validation dataset, our empirical results show consistent improvements over the baselines across all scenarios (cf. Table 3).
SYNTHIA as the training set. Due to space constraints, we limited our study of SaN in the main paper to the scenario of using GTA as the source data (ef. Table 2 Here, we extend this study by training our models on SYNTHIA instead. Table reports the segmentation accuracy (in terms of loU) and the expected calibration error (ECE) for this case. Notably, SaN can still provide benefits for the expected segmentation accuracy if p-BN (í. e., using target instance normalization statistics) fails to improve over the t-NN baseline, as is the case with ResNet-50 in Table 7 the results remain on par with the t-NN baseline even when p-BN is significantly worse than 1-BN. In regard to calibration quality, the results are consistent with our model
*We use Pillow library (https: / 1 lattps://githubbcoo wwwp://wwwooooooo//too/eaoooeeee DRPCC/isuuc httpp:/// http://hhhhb.ooi//ggagaaaaaaaaaa
http://plllowww///// readtheeoos..oo to implement photometric augmentation.
18

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Source: GTA
Source: SYNTHIA
40
loU (%)
30
20
Dash
BDD
CS
IDD
10
0.0
0.1
0.3
0.4
0.5 0.6
1.0
0.3
0.4
0.5
0.6
0.7
0.9
10
Figure 5: Mean IoU (%, t) using SaN based on the optimal alpha on the development set (WildDash). We report scores for the target domains (Cityscapes, BDD, IDD) for the ResNet-50 backbone after training on GTA (left) and SYNTHIA (right).
Table 6: Mean ToU (%, t) comparison of SaN to allernalives: SN
[Tuo et al. 2019 and BIN
Nam Kim
2018).
Method
CS
BDD
IDD
Mean
SN
31.75
33.60
31.60
32.32
BIN
34.57
32.68
30.22
32.49
SaN (ours)
37.54
32.79
34.21
34.85
trained on GTA (ef. Table b)): Not only does SaN improve the prediction calibration of the baseline, it
again exhibits complementary effect with Monte Carlo dropout (Gal & Ghahramani 2016). Overall, the combined results from Table 2 and Table 7 demonstrate that SaN improves both the model accuracy and the calibration quality of the predictions in the out-of-distribution setting irrespective of the backbone network and specifics of the source data.
Comparison to other related work. We additionally compare SaN to alternative normalization strategies proposed the literature: Batch-Instance Normalization (BIN; Nam Kim 2018) and Switchable Normalization (SN: Luo al. 2019). Although both of these techniques share some similaritics with our SaN, these approaches were developed for different purposes. SN only shown improve in-domain accuracy, while BIN tackles domain adaptation for image classification, not domain generalization studied in this work. Furthermore, both methods modify the model architecture before training, while SaN works with any pretrained semantic segmentation model. Nevertheless, we implemented both SN and BIN in our segmentation model based on the ResNet-50 backbone. We trained these approaches on GTA in an identical setup as SaN. From results in Table we observe that SaN outperforms both BIN and by a significant margin in terms of mean IoU of the larget domains.
B.2 Further design choices and runtime analysis
Selecting parameters for self-adaptation. In self-adaptation, we only update the model parameters in the layers conv4_x, conv5_x, and the classification head for the ResNet-50 backbone. Due to the higher computational cost of the ResNet-101 backbone, only adapt the layers conv5_ and the classification head in this case. Note that we analyze the runtime costs for alternative configurations in Fig. 2 in the main text, as well as Fig. below.
Threshold and learning rate. We investigate the influence of hyperparameters for self-adaptation: threshold s and lcarning rate n. Table g reports the IoU after self-adaptation evaluated on the target domain Cityscapes after training with ResNet-50 backbone on GTA. As already observed for (cf. Fig. 5 self-adaptation is robust to the choice of hyperparameters.
19

---

Published in Transactions on Machine Learning Research (07/2023)
Table 7: (a) Segmentation accuracy using SaN. We report the mean loU(%, 1) on three target domains (Cityscapes, BDD, IDD) across both backbones. In contrast Table [ we trained the networks on SYNTHIA in both cases. As before, t-BN denotes train BN (Loffe & Szegedy 2015), while p-BN refers prediction-tine BN (Nado et al., 2 (b) ECE for Sa.V and MC-Dropout (Gal & Ghahramani 2016). We report ECE scores (%,, 4) for three target domains (Cityscapes, BDD, IDD) across both backbones.
(a)
(b)
IoU (%, t)
ECE (%, 4
Method
Method
CS
BDD
IDD
Mean
CS
BDD
IDD
Mean
ResNet-50
ResNet-50
37.50
43.19
40.11
40.26
w/ t-BN
31.83
24.30
24.73
26.95
w/ SaN
30.96
33.27
36.31
33.51
w/ p-BN
33.83
23.36
23.39
26.86
w/ MC-Dropout
34.82
37.30
36.63
36.25
w/ SaN (Ours)
36.14
26.66
26.37
29.72
w/ both (Ours)
30.66
33.06
35.60
33.11
ResNet-101
ResNet-101
31.39
33.77
36.56
33.91
W/ t-N
37.25
29.32
27.19
31.25
w/ SaN
30.33
31.83
36.26
32.81
W/ p-BN
34.58
24.24
22.32
27.05
w/ MC-Dropout
32.73
32.76
34.07
33.19
w/ SaN (Ours)
38.01
28.66
27.28 31.32
w/ both (Ours)
27.71
30.48
32.67 30.29
Table 8: Mean IoU (%, t) for SaN and MC-Dropout Gal & Ghahramani 2016). We report scores for three target domains (Cityscapes, BDD, IDD) on the ResNet-50 (He et al.) (016)) backbone trained on GTA. We provided the ECE results in Table 2b) to motivate self-adaptive learning approach, which relies
improved model calibration. SaN also consistently improves the loU of the MC-Dropout approach.
loU (%, 1)
Method
CS
BDD
IDD
Mcan
ResNet-50
30.95
28.52
32.78
30.75
w/ SaN (Ours)
37.54
32.79
34.21
34.85
w MC-Dropout
30.45
31.96
32.50
31.63
both (Ours)
38.84
35.13
35.55
36.50
Table 9: Mean IoU (%, t) for hyperparameter variation of threshold & and learning rate 17. We report scores the ResNet-50 backbone trained on GTA evaluated Cityscapes for solf-adaptation and SaN for reference.
Self-Adaptation (Ours): variation of 1/ (n 0.05, N₂
10)
w/ SaN (Ours)
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
ResNet-50
37.54
44.00
44.00
44.01
44.04
44.24
44.54
44.91
45.13
44.59
43.62
39.40
Self-Adaptation (Ours):
variation of v) ( = 0.7, N₁ = 10)
w/ SaN (Ours)
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09 0.1
ResNet-50
37.54
44.40
44.77 44.95
45.06
45.13
45.08
45.06 45.07 45.04
44.96
The choice of augmentation strategies. We verify the influence of the augmentation type used by self-adaptation. Recall from Sec. 5.2 that we use multiple scales with horizontal flipping and grayscaling to augment one image sample. We compare flipping-oni only, scaling-only, and grayscaling-only version of our self-adaptation to the combination of flipping, grayscaling, and the spatial scaling, which we used in the main lext. We used a ResNet-50 backbone trained on GTA and report the accuracy on Cityscapes in Table [0] We observe a significant boost in accuracy in comparison to a strong baseline that uses our SaN with no augmentations. Furthermore, we show that using multiple scales more important than flipping for self-adaptation. Note that the augmentations used do not impact the runtime in a significant way, since the batch sizes between these setups vary insignificantly; it is the backpropagation that dominates the main
20

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Table 10: The role of the augmentation type in self-adaptation, We report mean loU (%, f) and runtime (ms, 4) for TTA (Simonyan & Zisserman, 2015 and our celf-adaptation for the GTA source domain and the Cityscapes target domain for the ResNet-50 backbone.
TTA
Ours
Method
IoU
Runtime
IoU
Runtime
Baseline
30.95
314
31.47
7135
SaN
37.54
314
39.04
7135
Multiple scales
42.27
749
44.92
7272
Horizontal flipping
38.01
986
39.33
7593
Grayscaling
37.96
908
39.65
7193
Multiple scales + horizontal flipping
42.48
1316
44.94
7890
Multiple scales + grayscaling
42.28
1202
45.06
7486
Multiple scales + horizontal flipping + grayscaling
42.56
1308
45.13
7862
Table 11: Runtime with TTA Simonyaan Zisserman, 2015) or self-adaptation. We report the runtime (ms, 1) for both ResNet-50 and ResNet-101 on three dominant resolutions of 2048 1024, 1280 x 720, and 1920 x 1080, corresponding to the target domains Cityscapes, BDD, and IDD, respectively.
Input resolution
Method
CS
BDD
IDD
Mean
ResNet-50 (w/ SaN)
314
136
214
221
TTA (w/ SaN)
1308
713
766
929
Self-adaptation (ours)
7862
3742
5307
5637
ResNet-101 (w/ SaN)
458
239
252
316
TTA (w/ SaN)
1519
766
860
1048
Self-adaptation (ours)
9060
4241
6142
6481
computational footprint. Varying the number of iterations, as studied in Sec. 5.1 provides a more flexible mechanism for accuracy-runtime tradc-off.
Inference time. Table compares the inference time of our self-adaptation w. r. t. test-lime augment ation (TTA) and single-scale inference across range of input resolutions available the target domains. We obtain these results by rumning inference single NVIDIA GeForce RTX 2080 GPU. To improve the runtime estimate, for each dataset average the inference time over the complete image set. This is to account for small deviations in the input resolution (e. 9., IDD mostly contains images of resolution 1920 x 1080, but also has images with resolution 1280 x 720). Since our self-adaptation uses 10 update iterations, the increase in the inference time w. r. t. TTA is expected. Although such cost may be detrimental for real-time applications, the significant accuracy benefits of self-adaptation (ef. Table 4 may find appeal applications where the importance of the prediction quality outweighs the incurred overhead in the frame rate (e. 9., medical image analysis). Furthermore, self-adaption can amortize the runtime costs by using fewer update iterations, while still providing clear accuracy improvement. For example, using N, 3 iterations decreases the runtime of self-adaptation almost twofold while preserving around 95% of the model accuracy attainable with more iterations.
We also investigated the influence using the automatic mixed precision module PyTorrh and its influence on the inference runtime. While maintaining an identical loU on Cityscapes using the ResNet-50 backbone, mixed precision with N₂ = 10 reduced the runtime by almost 30%: the inference takes only 5746 ms compared to 7862 using single precision.
21

---

Published in Transactions on Machine Learning Research (07/2023)
Table 12: Mean IoU (%. 1) using our self-adaptation integrated with DeepLabu3+ (Chen et al. 2018b based on a ResNet-50 and ResNet-101 backbone as well with HRNet- W18, HRNet- W48 (Wang et ol. 202161, and
UPerNet [Niao et el. 2018) with a Swin-T T backbone (Liu al. 2021). We observe substantial improvements of the segmentation accuracy all three target domains (Cityscapes, BDD, and IDD) after training GTA.
Target domains
Method
CS
BDD
IDD
Mean
+ ResNet-50 (Baseline)
37.51
35.45
37.50
36.82
Self-adaptation (ours)
46.56
43.17
44.07
44.60
DeepLabv3+ ResNet-101 (Baseline)
38.19
37.05
38.22
37.82
Self-adaptation (ours)
48.14
44.52
45.72
46.13
HRRNetWW15 (Baselinc)
33.08
29.40
32.97
31.82
Self-adaplation (ours)
44.05
38.29
43.78
42.04
HRNett (Baseline)
34.66
30.85
34.64
33.38
Self-adaptation (ours)
48.82
42.79
43.74
45.12
UPerNet Swin-T (Baseline)
39.67
36.04
39.74
38.48
Self-adaptation (ours)
45.04
39.77
44.33
43.05
We additionally provide runtime-aceuracy plots for GTA + BDD and GTA + IDD generalization in Fig. 6 The data supports our conclusions drawn on GTA + Cityscapes generalization (ef. Sec. 5.3) that (i) self- adaptation provides clear advantages segmentation accuracy over basclines at reasonable increase of the inference time; (ii) it both more accurate and more elficient than model ensembles; and (tii it exhibits flexible auttme-accuuracy trade-off by means of varying the number of update iterations and the number of the layers to adjust.
B.3 Self-adaptation with state-of-the-- art architectures
Our approach generalizes to recent architectures. We trained five state-of-the-art segmentation models GTA: DecpLabv3- (Chen et al. 2018b) with both ResNet-50 and ResNet-101 backbone, HRNet- W18 and HRNet-Wal (Wang et al. 2021b), as well as UPerNet (Xiao et al., 2018) with a Swin-T Transformer backbone (Lu et al. 2021). Table 12 reports consistent and substantial improvement of the mean IoU over the baseline, across all these architectures and the target domains.
B.4 Evaluation on diverse domain shifts
The ACDC dataset (Sakaridis et al. 2021 offers densely labeled driving scenes under adverse weather conditions such as fog, rain, and snow. As far as we are aware, previous work on domain generalization for semantic segmentation did not consider this benchmark. Therefore, we are only able to compare our results to our own bascline.
We evaluate Deeplabv1 model with ResNet-50 backbone trained on GTA with and without our self- adaptation approach. Table 13 presents the results. Our method demonstrates a stark improvement across all weather conditions outperforming our baseline model by 13.57% on average. This experiment underscores strong multi-target domain generalization aeff-adaptation, since we evaluate the same model as we did for other target datasets without any change (ef. Table 4).
B.5 Qualitative examples
We provide additional qualitative results by running inference on three models: ResNet-101 trained on GTA in Fig. 7 ResNet-50 and ResNet-101 trained on SYNTHIA in Figs. and g respectively. Similar to our observations in Soc. 5.3 (ef. main text). our approach exhibits more homogeneous semantic masks with visibly fewer jagged-shaped artifacts than the baseline (e. g., "sidewalk" false positives in Fig. 7). Models with
22

---

Published in Transactions on Machine Learning Research (07 (07/2023)
TTA
+ TTA
IoU (%)
35
SaN
Ours (conv4_ x conv5_ x- Head)
SaN
Ours (conv5 x Head)
30
Baseline
Ours (Headd
Bascline
Model Ensemble
25
2
7
10
Runtime (s)
40-
TTA
TTA
(%)
SaN
SaN Baseline
Ours (conv4 x conv5 x Hcad)
loU
Bascline
Ours (conv5 Head)
30
Ours (Head)
Model Ensemble
25
10
Runtime
Figure 6: Runttmm--accurucy comparison on GTA + BDD (top) and GTA - IDD (bottom) generalization using ResNet-500. The curves on this plot trace self-adaptation iterations, i. the first point corresponds N₂ I, while the last shows N₂ 10. While self adlatatapt increases the inference time of the baseline and TTA for the sake of improved accuracy, it is still more efficient and accurate than model ensembles of 10 networks. The choice of the layers for self-adaplation updates (the naming follows He et al. 2016 further provides a favorable uuntie-accuraee trade-off. The runtime is computed on single NVIDIA GeForce RTX 2080 GPU.
Table 13: Mean IoU (%, 1) using self-adaptation integrated with DeepLabul based ResNet-50 backbone. We observe substantial improvements of the segmentation accuracy under adverse weather conditions (fog, rain, and snow) on the ACDC (Sakaridis et al. 2021) validation set after training on the synthetic GTA.
Weather conditions
Method
Fog
Rain
Snow
Moan
ResNet-50 (Baseline)
26.12
27.32
24.64
26.03
Self-adaptation (ours)
41.61
37.54
39.66
39.60
self-adaptation may still struggle with cases of mislabeling regions with an incorrect, but semantically related class. For example, the model often assigns "sidewalk" to the road pixels from BDD and IDD in Figs. S and g This is an expected outcome if the erroncous labels are already contained in the pscudo-labels of the initial prediction, on which self-adaptation relies. These failure cases occur more frequently if the domain shift between the train and the test distributions is more significant, such as between SYNTHIA and BDD, which can lead poorly calibrated predictions. Since applying SaN alone results in improved calibration (cf. cope well with milder domain shift scenarios,
Table b)),, it alleviates this issue, and hence self-adaptation as result. As the examples on Cityscapes and Mapillary in Figs. and g show, despite the baseline model exhibiting some degree of this failure mode, our inference method visibly rectifies these errors.
We additionally ran our inference on video sequences and include the results as part of our supplementary material. Confirming our previous analysis of the qualitative results (ef. Sec. 5.2 we observe that our approach clearly improves the segmentation quality and removes some of the most pathological failure modes of the baseline (e. 9-, the lower middle part of the frame area).
23

---

Published in Transactions on Machine Learning Research (07 (07)))
GTA +
Cityscapes
GTA - BDD
GTA + Mapillary
GTA 7 IDD
Imagee
Ground Truth
Baseline
Ours
Figure 7: Qualitative semantic segmentation results for generolization from GTA to Cityscapes, BDD, Mopillary, and IDD for the ResNet-101 backbone.
B.6 Failure Cases
Conceptually, if the initial semantic prediction is incorrect and confident, which may happen due to imperfect model calibration, such enror is likely to end up in the pseudo-labels and lead astray the self-adaptation process. The rightmost columns in Figs. [ and g already visualized this failure mode: the bascline assigns large areas of the "road" class the "sidowalk" category. We have further extended these examples with Fig. TO Misguided by incorrect pseudo-labels, our self-adaptation may exacerbate this issue, e., propagate the incorrect label to the areas sharing the same appearance. To gain further insights, we investigated this issuc statistically. The histograms in Fig. 11 show relative improvement of our self-adaptive inference strategy w. r. J. the baseline in terms of ToU on four validation sets. We conclude that such decrease in segmentation accuracy is actually quite rare: less than 10% of the images, on average, exhibit lower accuracy. In most such cases the accuracy reduces only marginally (by less than 5%), and only fraction of images less than on average deteriorate in accuracy by at most 10%. The overwhelming majority of the image samples benefit from our self-adaptive process, which increases their accuracy by up to 35% IoU compared to the baseline.
C Dataset details
GTA. GTA |Richter et al. 2016: is a street view datasot generated semi-automatically from the computer game Grand Theft Auto V. The dataset consists of 12,403 training images, 6,382 validation images, and 6,181 testing images of resolution 1914 x 1052 with 19 different. semantic classes.
SYNTHIA. We use the SYNTHIA-RAND CAT.AAAACCTTTCPEESSS subset of the synthetic dataset SYNTHIA Ros al. 2016). which contains 9,400 images, and has 16 semantic classes in common with GTA. Images have a resolution of 1280 x 760 pixels.
24

---

Published in Transactions on Machine Learning Research (07 (07/2023)
SANTHIA-Cityscapaa SYNTHIA-BDI SYNTHIA Mapillary
SYNTHIA-IID
Imagee
Ground Truth
Baseline
Ours
Figure &: Qualitative semantic segmentation results for generalization from SYNTHIA to Cityscapes, BDD, Mopillary, and IDD for the ResNet-50 backbone.
WildDash. The WildDash benchmark (Zendel al. 2018 was developed evaluate models regarding their robustness for driving scenarios under real-world conditions. It comprises 4,256 images of real-world scenes with resolution of 1920 X 1080 pixels.
Cityscapes. Cityscapes (Cordts et al. 2016) is an ego-centric street-scene dataset and contains 5,000 high-resolution images with 2048 x 1024 pixels. It split into 2,975 train, 500 val, and 1,525 test images with 19 semantic classes being annotated.
BDD. BDD (Yu et al., 2020) is a driving vidco datasct, which also contains semantic labelings with the idantical 19 classes in the other datasets. Images have resolution of 1280 720 pixels. The training, validation, and test sets contain 7,000, 1,000, and 2,000 images, respectively.
IDD. IDD Varma et al. 2019 is dataset for road seene understanding in unstructured environments. It contains 10,003 images annotated with 34 classes even though we only evaluate on the 19 classes overlapping with the other datasets. IDD is split into 6,993 training images, 981 validation images, and 2,029 test images.
Mapillary. Annotations from Mapillary (Neuhold al. 2017) contain 66 object classes; analogously IDD only evaluate the 19 classes overlapping with the other datasets. The dataset is split into training set. with 18,000 images and validation set with 2,000 images with minimum resolution of 1920 1080 pixels.
25

---

Published in Transactions on Machine Learning Research (07 2023)
MUNTHIA--Ctyssae SYNTHIA-BDI SYNTHIA HANTHA--Mapillay
SYNTHIA- -ID
Image
STOP
Ground Truth
Baseline
Ours
Figure 9: Qualitative semantic segmentation results for generalization from SYNTHIA Cityscapes, BDD,
Mopillary, and IDD for the ResNet-101 backbone.
GTA + Cityscapes
GTA - BDD
GTA 7 Mapillary
GTA - IDD
Image
Ground Truth
Baseline
Ours
Figure 10: Failure cases of semantic segmentation for generolization results from GTA Cityscapes, BDD, Mopillary, and IDD for the ResNet-50 backbone.
26

---

Published in Transactions on Machine Learning Research (07/2023)
CS
BDD
IDD
Mapillary
40
Amount of images (%)
30
20
10
10 O 10 20 30
40 -10 O 10 20
30 40
-10 0 10
20 30
40
- 10 O 10
20 30
40
IoU (%)
Figure 11: Empirical distribution of the loU change of individual images for generalization from source domain (GTA) to target domains (Cityscapes, BDD, IDD and Mapillary) for the ResNet-50 backbone. We visualize the relative improvement of our self-adaptive inference strategy u. r. the baseline in terms of accuracy with IoU (%).
27


Reviews:
[{"review_id": "VqTf0pCp5o", "reviewer": "Reviewer_ycjh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- This paper is well-organized. The writing is of high quality.\n- This problem is important, and the proposed method shows strong empirical performance.\n- The experimental results are extensive.\n\nWeaknesses:\n\n- In my opinion, the claim 'Enhancing Generalization with a Single Sample' is not sufficiently convincing. The proposed method introduces many tunable hyper-parameters. For example, in the fine-tuning process and adjusting the statistics of BN layers. The values of these hyper-parameters need to be determined in the validation set. Hence, more exactly, more than a single sample is leveraged in this process.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an approach for domain-adaptive semantic segmentation. The basic assumption behind this setting is that the distribution and the training and validation data are not the same, which is of practical importance. The authors propose a self-adaptive algorithm that consists of two major components. First, it fine-tunes some convolutional layers using the test inputs with consistency regularization. Second, it interpolates between the source and the target statistics in Batch Normalization layers. Experimental results validate the effectiveness of this approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the Weaknesses above. I think the authors should clarify their claims on  'Enhancing Generalization with a Single Sample' .", "broader_impact_concerns": "I have no concerns about the broader impacts and the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tFBa5AFlK7", "reviewer": "Reviewer_eXNC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n----------------\n1. Novel approach: The authors propose a self-adaptive approach for semantic segmentation that adjusts the inference procedure for each sample, which is a unique and innovative approach to address the generalization problem from synthetic data. This can potentially lead to improved performance in semantic segmentation tasks.\n\n2. Rigorous evaluation protocol: The authors highlight the lack of systematic evaluation in existing works and propose a comprehensive evaluation protocol that includes the use of multiple domains in the test set, employing a single model for all test points, independently selecting the model regardless of the test samples, and clearly specifying the validation data. This rigorous evaluation protocol enhances the reliability of the research findings.\n\n3. Empirical evaluation: The authors conduct a thorough empirical evaluation by comparing their proposed method with several state-of-the-art baselines on multiple datasets and models. This extensive evaluation provides evidence of the effectiveness of their approach and adds credibility to their findings.\n\n\nWeaknesses\n---------------\n1. Although the idea to update model during inference is interesting, it is going to be make the inference procedure computationally expensive. Usually inference is expected to be very fast.\n\n2. Lack of reasoning or rationale behind the method. In particular why should the proposed method work well?  \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "\nIn this study, the authors study the generalization in semantic segmentation with synthetic data using adaptation techniques. They propose a self-adaptive approach that adjusts the inference procedure for each individual test sample. This approach involves creating a mini-batch from the given test sample using data augmentation techniques, averaging the softmax probabilities, and generating a pseudo label based on a class-dependent confidence threshold. The model parameters are then updated by minimizing the cross-entropy loss with respect to the pseudo label. This process is repeated for a few iterations before producing the final prediction and discarding the updated model after prediction.\nThe authors also highlight the lack of systematic evaluation in existing studies and propose steps for a rigorous and systematic evaluation of domain generalization methods for semantic segmentation. Their proposal includes using multiple domains in the test set, employing a single model for all test points, independently selecting the model regardless of the test samples, and clearly specifying the validation data.\nFollowing the revised evaluation protocol, the authors conduct a comprehensive empirical evaluation by comparing their proposed method with several state-of-the-art baselines on multiple datasets and models. They observe that their proposed method outperforms the baselines.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Could you please summarize the method in an algorithm block?\n\n2. Could you please provide reasoning behind the method’s effectiveness. It could be either via some controlled synthetic experiments or even better with some theoretical analysis.\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aRFOrTKF1d", "reviewer": "Reviewer_TqUP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n* Paper is well written.\n* Authors provide comprehensive experiments. \n\nWeakness:\n* Under which scenario(s) would we like to pay this computation/time cost for single image adaptation? Examples in Figure 1 are from driving scenes. I suppose driving scene understanding is highly latency-sensitive.\n* Furthermore, making predictions on a batch of samples can speed up the inference time. Will this method be adopted for multi-sample adaptation during inference? If so, will any study about the impact of inference batch size on mIoU be interesting?\n* As discussed in the abstract, two core parts of the method is previously proposed. I hope authors could better explain/claim their core novelty.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Deep networks for semantic segmentation lack out-of-domain generalization. In this work, a self-adaptive approach is proposed to adjust the inference process for each input sample, fine-tuning parameters, and interpolating in Batch Normalization layers. This approach sets a new state-of-the-art accuracy on generalization benchmarks, complementing model regularization for improving generalization to out-of-domain data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Introducing test-time adaptation (both parameter label and BN level) involves more hyperparameters to tune, like $\\psi$, augmentations, $N_t, \\eta$. In more diverse domain gaps (low resolution, weather changes, low light, etc.), the proposed method may not guarantee that it is robust to the tuning of these hyperparameters. I hope the authors could justify if the proposed method is user-friendly or any ablation study on the sensitivity of the method to these hyperparameters.", "broader_impact_concerns": "I think in general this work is interesting to the community of domain adaptation, domain transfer, and semantic segmentation.\nTest-time adaptation is definitely useful for solving domain gaps and perhaps synthetic-to-real generalization. However, computation and time costs could limit the possible use cases of this work. I hope authors could for example enumerate useful use cases, to verify the true broad impact of this work in the real world.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a self-adaptive semantic segmentation approach that enhances the generalization capability in the presence of domain shifts. The idea is both simple and effective, the paper is well-written, and the experimental verification is solid. Therefore, the AE recommends Accept with minor revision. 

The authors are asked to make two vital revisions to the paper organization: 1) Move the Algorithm Block (i.e., B.7 and Algorithm 1 in the Appendix) to the main body of the paper. 2) Provide a detailed analysis of the impact of hyperparameters and a formal guideline for tuning them in the main body of the paper.

Decision: Accept (poster)