Uncertainty Modeling for Out-of-Distribution Generalization
Authors: Xiaotong Li
Yongxing Dai
Yixiao Ge
Jun Liu
Ying Shan
LINGYU DUAN
Venue: ICLR 2022 Poster (2022)
Keywords: domain generalization
uncertainty modeling

Abstract: Though remarkable progress has been achieved in various vision tasks, deep neural networks still suffer obvious performance degradation when tested in out-of-distribution scenarios. We argue that the feature statistics (mean and standard deviation), which carry the domain characteristics of the training data, can be properly manipulated to improve the generalization ability of deep learning models. Common methods often consider the feature statistics as deterministic values measured from the learned features and do not explicitly consider the uncertain statistics discrepancy caused by potential domain shifts during testing. In this paper, we improve the network generalization ability by modeling the uncertainty of domain shifts with synthesized feature statistics during training. Specifically, we hypothesize that the feature statistic, after considering the potential uncertainties, follows a multivariate Gaussian distribution. Hence, each feature statistic is no longer a deterministic value, but a probabilistic point with diverse distribution possibilities. With the uncertain feature statistics, the models can be trained to alleviate the domain perturbations and achieve better robustness against potential domain shifts. Our method can be readily integrated into networks without additional parameters. Extensive experiments demonstrate that our proposed method consistently improves the network generalization ability on multiple vision tasks, including image classification, semantic segmentation, and instance retrieval.

Published as a conference paper at ICLR 2022
UNCERTAINTY MODELING FOR -UT-OF-DISTRIBUTION GENERALIZATION
Xiaotong Li', Yongxing Dai', Yixiao Ge Jun Liu', Ying Shan", Ling-Yo Yu Duan 1+ Peking University, Beijing, China AARC Lab, Tencent PCG "Singapore University of Technology and Design, Singapore 'Peng Cheng Laboratory, Shenzhen, China
lixiactengestu wwt.:esst..uuueeeeeee pku.edu.cn, (yongxingdai, lingyu]@pku.edi edu. {yixiacge, ttensen, com, jur. Jur.
ABSTRACT
Though remarkable progress has been achieved in various vision tasks, deep neu- ral networks still suffer obvious performance degradation when tested in out-of- distribution scenarios. We argue that the feature statisties (mean and standard deviation), which carry the domain characteristics of the training data, can be properly manipulated to improve the generalization ability of deep learning mod- els. Common methods often consider the feature statistics as deterministic values measured from the learned features and do not explicitly consider the uncertain statistics discrepancy caused by potential domain shifts during testing. In this pa- per, we improve the network generalization ability by modeling the uncertainty of domain shifts with synthesized feature statistics during training. Specifically, we hypothesize that the feature statistic, after considering the potential uncertain- ties, follows multivariate Gaussian distribution. Hence, each feature statistic is no longer a deterministic value, but probabilistic point with diverse distribution possibilities. With the uncertain feature statistics, the models be trained alleviate the domain perturbations and achieve better robustness against potential domain shifts. Our method can be readily integrated into networks without addi- tional parameters. Extensive experiments demonstrate that our proposed method consistently improves the network generalization ability on multiple vision tasks, including image classification, semantic segmentation, and instance retrieval. The code can be available [Iuns///ggtuubboom//riiooo//////
INTRODUCTION
Deep neural networks have shown impressive success in computer vision, but with a severe reliance on the assumption that the training and testing domains follow an independent and identical distri- bution (Ben-David et al. 2010 Vapnik 1992). This assumption, however, does not hold in many real-world applications. For instance, when employing segmentation models trained on sunny days for rainy and foggy environments (Choi et al. 2021). recognizing paintings with models that trained on photographs (Li et al,, 2017), inevitable performance drop can often be observed such -uitt--distributionn deployment scenarios. Therefore, the problem of domain generalization, aim- ing to improve the robustness of the network on various unseen testing domains, becomes quite important.
Previous works (Huang & Belongie 2017: Li et al. 2021 demonstrate that feature statistics (mean and standard deviation), as the moments the learned features, carry informative domain charac- teristics of the training data. Domain characteristics primarily refer to the information that is more specific the individual domains but less relevant the task objectives, such as the photo style and capturing environment information in object recognition. Consequently, domains with different data distributions generally have inconsistent feature statistics (Wang et al.) (20206) 2019a Gao al.) 202 Most deep learning methods follow Empirical Risk Minimization principle (Vapnik 1999) to minimize their average error over the training data (Shen et al.) 2021). Despite the satisfactory
"Corresoonding Author.

---

Published as a conference paper at ICLR 2022
different direction and interaity of offset
samples of original statstics
samples of synthetic statislics
potendall feature stutisliss distribution
original sample
original sample
synthatic samples
synthatic samples
Figure 1: The visualization of reconstructed samples with synthesized feature statistics, using pre-trained style transfer auto-encoder (Huang & Belongie 2017). The illustration of the feature statistics shifts, which may vary in both intensity and direction (i.e., different offsets in the vector space of feature statistics). We also show images of "new" domains generated by manipulating feature statistic shifts with different direction and intensity. Note these images for visualization only, rather than feeding into the network for training.
performance on the training domain, these methods do not explicitly consider the uncertain statis- ties discrepancy caused by potential domain shifts during testing. As result, the trained models tend overfit the training domain and show vulnerability to the statistic changes at testing time, substantially limiting the generalization ability of the learned representations.
Intuitively, the test domains may bring uncertain statistics shifts with different potential directions and intensities compared to the training domain (as shown in Figure []). implying the uncertain nature of domain shifts. Considering such "uncertainty" of potential domain shifts, synthesizing novel feature statistics variants to model diverse domain shifts can improve the robustness of the trained network to different testing distributions. Towards this end, we introduce a novel probabilis- tic method to improve the network generalization ability by properly modeling Domain Shifts with Uncertainty (DSU), i.e., characterizing the feature statistics as uncertain distributions.
In method, instead of treating each feature statistic as deterministic point measured from the feature, we hypothesize that the feature statistic, after considering potential uncertainties, follows a multi-variate Gaussian distribution. The distribution "center" is set each feature's original statistic value, and the distribution "scope" represents the variant intensity considering underlying domain shifts. Uncertainty estimation adopted here depict the distribution "scope" of probabilistic feature statistics. Specifically, we estimate the distribution "scope" based on the variances of the mini-bated statistics an efficient non-pan -orrammmetrrie manner. Subsequently, feature statisties variants are randomly sampled from the estimated Gaussian distribution and then used to replace the original deterministie values for modeling diverse domain shifts, as illustrated in Figure 2 Due to the gen- erated feature statistics with diverse distribution possibilities, the models can be trained to properly alleviate the domain perturbations and encode better domain-invariant features.
Our proposed method is simple yet fairly effective alleviate performance drop caused by domain shifts, and can be readily integrated into existing networks without bringing additional model param- eters or loss constraints. Comprehensive experiments the superiority our proposed method, indicating that introducing uncertainty to feature statistics can well improve models' generalization against domain shifts.
wide range of vision tasks demonstrate
RELATED WORK
2.1 DOMAIN GENERALIZATION
Domain generalization (DG) has been attracting increasing attention in the past few years, which aims to achieve -iitt---iitiiiutioo generalization on unseen target domains using only single or multiple source domain data for training (Blanchard et al. (2011)). Research on addressing this problem has been extensively conducted in the literature (Zhou et al. (2021a); Waan et al. (202T); Shen et al. (202T)). Here some studies that are more related to work introduced below.
2

---

Published as a conference paper at ICLR 2022
Data Augmentation: Data augmentation is effective manner for improving generalization abil- ity and relieving models from overfitting in training domains. Most augmentation methods adopt various transformations at the image level, such as AugMix (Hendrycks et al. (2020) and CutMix (unn et al. (2019)) Besides using handcraft transformations, mixup (Zhang et al. (2018) trains the model by using pair-wise linearly interpolated samples in both the image and label spaces. Mani- fold Mixup (Verma et al. (2019)) further adopts this linear interpolation from image level to feature level. Some recent works extend the above transformations to feature statisties for improving model generalization. MixStyle (Zhou et al. (202Tb)) adopts linear interpolation on feature statistics of two instances to generate synthesized samples. The PAdaIn (Nuriel al.) (202I)) swaps statisties between the samples applied with random permutation of the batch.
Invariant Representation Learning: The main idea of invariant representation learning is to enable models to learn features that are invariant to domain shifts. Domain alignment-based approaches (Li al. (2 (b) learn invariant features by minimizing the distances between different distributions. Instead of enforcing the entire features to be invariant, disentangled feature learning approaches Chattopadhyay et al. (2020); Piratla et al. (2020)) decouple the features into domain-specifie and parts and learn their representations simultaneously. In addition, normalization- based methods (Pan et al. (2018);(hoot et al. 2021)) can also be used to remove the style information to obtain invariant representations.
There are also some effective learning strategies that can be leveraged to improve generalization ability. Ensemble learning an effective technique in boosting model per- formance. The ensemble predictions using a collection of diverse models (Zhou et al. (2020b)) or modules (Seo al. (2020)) can be adopted improve generalization and robustness. Meta- barming-based methods (Finn et al. (2017); Li et al. (2018a); Dai et al.] (202T)) learn to simulate the domain shifts following an episode training paradigm. Besides, self-challenging methods, such as RSC (Huang et al. (2020)), force the model to learn general representation by discarding dominant features activated on the training data.
Learning Strategies:
2.2 UNCERTAINTY DEEP LEARNING
Uncertainty capturing the "noise" and "randomness" inherent in the data has received increasing attention in deep representation learning. Variational Auto-encoder (Kingma & Welling) (2013). as important method for learning generative models, can be regarded as method to model the data uncertainty in the hidden space. Dropout (Srivastava et al. (2014), which is widely used in many deep learning models to avoid over-fitting, can be interpreted to represent model uncertainty as a Bayesian approximation (Gal & Ghahramani (2016)). In some works, uncertainty is used to address the issues of low-quality training data. In person re-identification, DistributionNet (Gal & Ghahramani (0166) adopts uncertainty to model the person images of noise-labels and outliers. In face recognition, DUL (Chang al.) (2020) and PFE ((Shi & Jain 2019)) apply data uncertainty to simultaneously learn the feature embedding and its uncertainty, where the uncertainty is learned through a learnable subnetwork describe the quality of the image. Different from the aforemen- tioned works, our proposed method is used to model the feature statistics uncertainty under potential domain shifts and acts as a feature augmentation method for handling -ustr-ddistrbbttion general- ization problem.
3 METHOD
3.1 PRELIMINARIES
Given & E BBxCxHXW to be the encoded features in the intermediate layers of the network, we denote μ E BBC and a BBC as the channel-wis wise feature mean and standard deviation each instance in mini-batch, respectively, which can be formulated as:
H
W
m(x) =
Σ
Db.e.b,w
(1)
HW
A=
w=1
H
W
x²(x)
0²
(Tb a(x))²
(2)
HW
h= W
3

---

Published as a conference paper at ICLR 2022
infermediatle feature
statistics
batchsize
treining imaged
original feature statislis (1A)
#' ~ Ml,
Randamly Sampling
Uncartaiity Estimation E
Figure 2: Illustration of the proposed method. Feature statistic is assumed to follow Gaussian distribution during training. When passed through this module, the new feature statistics randomly drawn from the corresponding distribution will replace the original ones to model the diverse domain shifts.
As the abstraction of features, feature statistics can capture informative characteristies of the cor- responding domain (such as color, texture, and contrast), according to previous works (Huang & Belongie 2017; Li al. 202T). In out-of-distribution scenarios, the feature statistics often show inconsistency with training domain due to different domain characteristies (Wang et al.) 2019a Gao al. 2021a), which is ill-suited deep learning modules like nonlinearity layers normalization layer and degenerates the model's generalization ability (Wang et al.] (020b). However, most of the deep learning methods only treat feature statistics as deterministic values measured from the fea- tures while lacking explicit consideration of the potential uncertain statistical discrepancy. Owing to the model's inherent vulnerability to such discrepancy, the generalization ability the learned representations is limited. Some recent methods (Nuriel al. 2021 Zhou et al., (021b) utilize fea- ture statistics to tackle the domain generalization problem. Despite the success, they typically adopt linear manipulation (i.e., exchange and interpolation) on pairwise samples generate statistics, which limits the diversity of synthetic changes. Specifically, the direction their variants determined by the chosen reference sample and such internal operation restricts their variant in- tensity. Thus these methods are sub-optimal when handling the diverse and uncertain domain shifts in real world.
feature
3.2 MODELING DOMAIN SHIFTS WITH UNCERTAINTY
Given the arbitrary testing domains with uncertain feature statistic shifts in both direction and inten- sity, properly modeling the domain shifts becomes an important task for tackling the challenge of domain generalization problem.
Considering the uncertainty and randomness of domain shifts, it is promising to employ the meth- ods of "uncertainty" to treat the "uncertainty" of domain shifts. In this paper, we propose novel method by modeling Domain Shifts with Uncertainty (DSU). Instead treating each feature statis- tic deterministic value measured from the learned feature, we hypothesize that the distribu- tion of each feature statistic, after considering potential uncertainties, follows a multi-variate Gaus- sian distribution. This means each feature statistic has probabilistic representation drawn from a certain distribution, i.e., the feature statistics mean and standard deviation follow N(,, )3)) and N(o, >) respectively. Specifically, the corresponding Gaussian distribution's 's center is set each feature's 's original statistics, while the Gaussian distribution's 's standard deviation describes the uncer- tainty scope for different potential shifts. Through randomly sampling diverse synthesized feature statistics with the probabilistic approach, the models can be trained to improve the robustness of the network against statistics shifts.

---

Published as a conference paper at ICLR 2022
3.2.1 UNCERTAINTY ESTIMATION
Taking the uncertainty domain shifts into consideration, the uncertainty estimation in our method aims to depict the uncertainty scope of each probabilistic feature statistic. However, the testing domain is unknown, which makes it challenging obtain an appropriate variant range.
Some generative-based studies (Shen & Zhou 2021 Wang et al. 2019b) show that the variances between features contain implicit semantic meaning and the directions with larger variances can imply potentials of more valuable semantic changes. Inspired by this, we propose simple yet effective non-pan con-aammetrric method for uncertainty estimation, utilizing the variance of the feature statistics to provide some instructions:
(x)
(m(()) Bo(u(x)))²
(3)
B
(x)
/ E (olt))Bolo(<))) B
1
(4)
where En E IRC and Σ. E RCC represent the uncertainty estimation of the feature mean and feature standard deviation a. respectively. The magnitudes of uncertainty estimation can reveal the possibility that the corresponding channel may change potentially. Although the underlying distribution of the domain shifts is unpredictable, the uncertainty estimation captured from the mini- batch can provide an appropriate and meaningful variation range for each feature channel, which does not harm model training but can simulate diverse potential shifts.
3.2.2 PROBABILISTIC DISTRIBUTION OF FEATURE STATISTICS
Once the uncertainty estimation of each feature channel is obtained, the Gaussian distribution for probabilistic feature statistics can be established. To use randomness to model the uncertainty, we adopt the random sampling to further exploit the uncertainty the probabilistic representations. The new feature statistics, mean ((x) ~ N(u, Σ/) and standard deviation 7(x) ~ Na Σ can be randomly drawn from the corresponding distributions as:
8(x) = μ(z) + (x))
Ep ~ NO0,1).
(5)
>xx) = (xx) t (x),
(0,
(6)
Here we use the ce-parameterizatioo trick Kingmma & Welling (2013) to make the sampling opera- tion differentiable, and e,, and e, both follow the standard Gaussian distribution. By exploiting the given Gaussian distribution, random sampling can generate various new feature statistics informa- tion with different combinations of directions and intensities.
3.2.3 IMPLEMENTATION
The implementation of our method by the means of AdaIN (Huang & Belongie (017)) and replaces the feature statistics with the randomly drawing ones to achieve the transformation. The final form the proposed method can be formulated as:
m(xx) - + (u(x) + (x))
DSU(#) (a(x) (x))
(xx)
7(m)
((x)
The above operation can be integrated at various positions of the network as a flexible module. Note that the module only works during model training and can be discarded while testing. To trade off the strength of this module, we set a hyperparameter p that denotes the probability to apply it. The algorithm is described in the Appendix. Benefiting from the proposed method, the model trained with uncertain feature statistics will gain better robustness against potential statistics shifts, and thus acquires better generalization ability.
5

---

Published as a conference paper at ICLR 2022
4 EXPERIMENTS
In order to verify the effectiveness of the proposed method improving the generalization ability of networks, we conduct the experiments a wide range of tasks, including image classification, semantic segmentation, instance retrieval, and robustness towards corruptions, where the training and testing sets have different cases of distribution shifts, such as style shift, synthetic-to-real gap, scenes change, and pixel-level corruption.
4.1 GENERALIZATION ON MULTI-DOMAL CLASSIFICATION
Setup and Implementation Details: We evaluate the proposed method on PACS (Li al. 2017 widely-used benchmark for domain generalization with four different styles: Art Painting, Cartoon, Photo, and Sketch. The implementation follows the official setup of MixStyle (Zhou et al. (b021b) with a leave-one-domule out protocol and ResNet18 (He et al. 2016) is used as the backbone. The random shuffle version of MixStyle is adopted for fair comparisons, which does labels. In addition to PACS, we also employ Office-Home Venkateswara et al. 2017) for multi- domain generalization experiments in the Appendix.
domain
The experiments results, shown in Table ! demonstrate our significant im- provement over the baseline method, which shows our superiority the conventional deterministic approach. Especially in Art and Sketch, our method has nearly 10% improvement in average ac- curacy. Furthermore, our method also outperforms the competing methods, which indicates our method that models diverse uncertain shifts on feature statistics is effective to improve network generalization ability against different domain shifts. Photo has similiar domain characteristics as ImageNet dataset and the slight drop might be due to the ImageNet pretraining (also discussed in (Xu et al., 2021)). Our DSU augments the features and enlarges the diversity of the training do- pre-trained knowledge from ImageNet thus
Experiment Results:
mains. In contrast, the baseline method preserves tends to overfit the Photo style dataset benefiting from pretraining.
Table Experiment results of PACS multi-domain classification task, RSC* denotes the reproduced
results from PAdaIN (Nuriel et al. 2021).
Method
Reference
Art
Cartoon
Photo
Sketch
Average (%)
Baseline
74.3
76.7
96.4
68.7
79.0
Mixup /Zhan et al.
2018
ICLR 2018
76.8
74.9
95.8
66.6
78.5
Manifold Mixup Verma
2019
ICMI 2019
75.6
70.1
93.5
65.4
76.2
CutMix Yun al.
2019)
ICCV 2019
74.6
71.8
95.6
65.3
RSC* Huang ct al
2020
ECCV 2020
78.9
76.9
94.1
76.8
81.7
L2A-O1 Zhou et al
2020a
ECCV 2020
83.3
78.2
96.2
73.6
82.8
SagNet
Nam et al.
2021
CVPR 2021
83.6
77.7
95.5
76.3
83.3
pAdaIN
Nuriel
2021
CVPR
81.7
76.6
96.3
75.1
MixStyle (Zhou ct al.
2021b
ICLR 2021
82.3
79.0
96.3
73.8
82.8
DSU
Ours
83.6
79.6
95.8
77.6
84.1
4.2
GENERALIZATION ON SEMANTIC SEGMENTATION
Semantic segmentation, as a fundamental application for au- tomatic driving, encounters severe performance declines due scenarios differences (Wang al. 2020a). GTA5 (Richter et al. 2016) is a synthetic dataset generated from Grand Theft Auto 5 game engine, while Cityscapes (Cordts al. 2016) real-world dataset collected from different cities in primarily Germany. To evaluate the cross-scenario generalization ability of segmentation models, we adopt synthetic GTAS for training while using real CityScapes for testing. The experiments are conducted on FADA released codes (Wang et al. (2020a)), using Deepl ab-v2 (Chen et al.) 2018) segmentation network with ResNet101 backbone. Mean Intersection over Union (mIOU) and mean Accuracy (mAcc) of all object categories are used for evaluation.
Setup and Implementation Details:
Table 2: Experiment results of semantic segmentation from synthetic GAT5 to real Cityscapes.
Method
Reference
mIOU (%)
mAce (%)
Baseline
37.0
51.5
pAdaIN Nuriel et al.
2021
CVPR 2021
38.3
52.1
Mixstyle Zhou et 202lb
ICLR 2021
40.3
53.8
DSU
Ours
43.1
57.0

---

Published as a conference paper at ICLR 2022
Experiment Results: Table 2 shows the experiment results compared to related methods. As for a pixel-level classification task, improper changes of feature statistics might constrain the perfor- mances. The variants generated from method centered on the original feature statistics with different perturbations. These changes of feature statistics are mild for preserving the detailed infor- mation in these dense tasks. Meanwhile, our method can take full of the diverse driving scenes and generates diverse variants, thus show significant improvement on mIOU and mAC by 6.1 6.1% and 5.5%, respectively. The visualization result is shown Figure
Unseen Domain
Baseline
Ours
Ground Truth
Figure 3: The visualization unseen domain Cityscapes with the model trained on synthetic GTA5.
4.3 GENERALIZATION ON INSTANCE RETRIEVAL
Setup and Implementation Details: In this section, person re-identification (ReID), which aims at matching the same person across disjoint camera views, is used to verify the effectiveness of our method on the instance retrieval task. Experiments are conducted on the widely used DukeMTMC Riistani al. (2016) and Market1501 (Zheng al. (2015)) datasets. The implementation is based on MMT (Ge et al. 2020) released codes and ResNet50 is adopted as the backbone. Meanwhile, mean Average Precision (mAP) and Rank-1 (R1) precision are used as the evaluation criterions.
Table 3: Experiment results of instance retrieval on RelD dataset DukeMTMC and Market A
+ denotes models are trained on while evaluated on B. For fair comparisons, we reproduce the experiments under the same framework.
Market + Duke
Duke - Market
Method
Reference
mAP (%)
RI (%)
mAP (%)
RI (%)
Baseline
25.8
42.3
26.7
54.7
PAdaIN (Nuriel et al. 2021
CVPR 2021
28.0
46.1
27.9
56.1
MixStyle Zhou al. 2021b)
ICLR 2021
28.2
46.7
28.1
56.6
DSU
Ours
32.0
52.0
32.4
63.7
RelD a fine- fineggrined instance retrieval task, where the subtle information of persons is important for retrieving an instance. MixStyle and pAdain rely on reference sample to generate new feature statistics, which might introduce confounded information from the reference sample. Compared to them, our method does better in maintaining the original information and also has more variant possibilities. The experiment results are demonstrated Table 3 Our method achieves huge improvement compared to the baseline method and also outperform MixStyle and PAdaIN by big margin.
Experiment Results:
4.4 ROBUSTNESS TOWARDS CORRUPTIONS
Setup and Implementation Details: We validate the proposed method for robustness towards cor- ruptions on ImageNet-C (Hendrycks & Dietterich (2019), which contains 15 different pixel-level corruptions. ResNet50 is trained with 100 epochs for convergence on large-scale ImageNet- K (Deng et al. (2009)) and the hyperparameter is set as 0.1 for training in ImageNet. We also add our method on APR (Chen et al. (202T)) a recently state-of-the-art method on ImageNet-C. that our method can be compatible with other image-level augmentation methods. Error is adopted as the evalution metric for clean ImageNet. Mean Corruption Error (mCE) is adopted as evaluation metric for ImageNet-C. which is computed the average of the 15 different corruption errors and normalized by the corruption error of AlexNet (Krizhevsky et al. (2012)).
verify
Experiment Results: Although the corruptions imposed the pixel level. they still introduce shift in the statistics (Benz et al.) (02I). So our method shows consistent improvement on ImageNet- C. Meanwhile, the instances in the testing may not always fall into the distribution of the training

---

Published as a conference paper at ICLR 2022
Table 4: Experiment results of clean image classification on ImageNet, and the robustness toward corruptions on ImageNet-C.
Clean
Compued (.)
Noti
Mior
Weather
Diginl
Faree CR)
Star
Repaoree
Clbss
Waaton
Zoum
Rnghi
Cuntear
Flaarc
Piaal
K1
75
APR-1XU
2N7
set, and they still have slight statistic shifts (Gao al. (2021b)) Thus can be seen that the within- dataset ImageNet accuracy is also increased. When combining APR with our method, mCE can be decreased from 65.0 % 64.1%, showing that method can be compatible with state-of-the-an methods on ImageNet-C for further improvement.
ABLATION STUDY
In this section, we perform an extensive ablation study of the proposed method on PACS and seg- mentation task (GTA5 to Cityscapes) with models trained on ResNet. The effects of different serted positions and hyper-paramete parameter of the proposed method are analyzed below. Meantime, we also analyze the effects on different choices of uncertainty distribution.
Effects of Different Inserted Positions: DSU can be plug- and-play module to be readily inserted at any position. Here
Table 5: Effects of different inserted positions.
Inserted Positions
Baseline
0-3
1-4
2-5
0-5
PACS
79.0
82,2
83.1
83.5
84.1
name the positions of ResNet after first Conv, Max Pooling
GTA5 to Cityscapes
37.0
41.1
40.9
42.1
43.1
layer, 1,2,3,4-th ConvBlock as 0,1,2,3,4.5 respectively. As shown in Table [1] matter where the modules are inserted, the performances are consistently higher than the baseline method. The results show that inserting the modules at positions 0-5 would have better performances, which also indi- cates modeling the uncertainty in all training stages will have better effects. Based on the analysis, we plug the module into positions 0-5 in all experiments.
Effects of Hyper-pan pprrreaaameter: The hyper-parameter of the probability p is to trade off the strength of feature statistics augmentation. As shown in Figure the results are sensitive to the probability setting and the accu- racy reaches the best results when setting p as 0.5, which also adopted as the default setting in all experiments if not specified.
RACS
GTAS Cityecapes
Dura
64
Figure 4: The effects on the hyper- parameter probability.
Choices of Uncertainty Distribution: In our method, the Gaussian distribution with uncertainty estimation is
adopted as the default setting, we also conduct other distributions for comparisons in Table 6 Specifically. Random denotes directly adding random shifts draw from fixed Gaussian (o0, 1), and Uniform denotes that the shifts are drawn from U(-E, E), where Σ is the scope obtained from uncertainty estimation. As variant scope will harm the model performances, indicating the variant range of feature statistics should have some instructions. Further analysis about different vanilla Gaussian distributions with pre-defined standard deviation are conducted in the Appendix. Meanwhile, the result of Uniform shows some improvement but is still lower than DSU, which indicates the boundless Gaussian dis- tribution is more helpful to model more diverse variants.
see, directly using Gaussian distribution with the improper
Table 6: Different choices of distribution for uncertainty.
Choice
Baseline
Random
Uniform
DSU
PACS
79.0
76.9
81.9
84.1
GTAS to Cityscapes
37.0
38.2
41.6
43.1
8

---

Published as a conference paper at ICLR 2022
6
FURTHER ANALYSIS
6.1 QUANTITATIVE ANALYSIS ON THE PROPOSED METHOD
In this subsection, will analyze the effects of the proposed method both intermediate features and feature representations. Quantitative experiments are conducted on PACS, where we choose Art Painting as the unseen testing domain and the rests are used as training domains.
To study the phenomena of feature statistic shifts, we capture the intermediate features after the second block in ResNet18 and measure the average feature statistics values of one category in the training and testing domain, respectively. The distributions of feature statistics are shown in Figure 5 As the previous works [Wang et al. 2020bb (019a) show, the feature statistics extracted from the baseline model show obvious shift due to different data distribution. It can be seen that the model trained with our method has less shift. Our method can help the model gain robustness towards domain shifts, as it properly models the potential feature statistic shifts.
Daneline
Banekine
feature mean
feature standard deviation
Figure 5: Quantitative analysis the shifts of feature statistics (mean and standard deviation) be- tween training source domains and unseen testing domain.
6.2 VISUALIZATION ON THE SYNTHETIC CHANGES
Besides the quantitative experiment results, we also obtain more intuitional view of the diverse changes provided by our method, through visualizing the reconstruction results using predefined autoencoder (Huang & Belongic (2017)), where the proposed module is inserted into the encoder, and inverse the feature representations into synthetic images after the decoder. As the results shown Figure 6 the reconstructed images obtained from our probabilistic approach show diverse syn- thetic changes, such the environment, object texture, and contrast, etc.
Figure 6: The visualization on diverse synthetic changes obtained from our method.
7 CONCLUSIONS
In this paper, we propose a probabilistic approach to improve the network generalization ability by modeling the uncertainty of domain shifts with synthesized feature statisties during training. Each feature statistic is hypothesized to follow a multi-variate Gaussian distribution for modeling the diverse potential shifts. Due the generated feature statistics with diverse distribution possibilities, the models can gain better robustness towards diverse domain shifts. Experiment results demonstrate the effectiveness of our method in improving the network generalization ability.
ACKNOWLEDGEMENT
This work supported by the National Natural Science Foundation of China under Grant 62088102, and part by the PKU-NTU Joint Research Institute (JRI) sponsored by a donation from the Ng Teng Fong Charitable Foundation.
The pretrained weights from http://grhhllcommmaaaoooooo/oaaa

---

Published as a conference paper at ICLR 2022
REFERENCES
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine learning, 79(1):151-175,
2010.
Philipp Benz, Chaoning Zhang, Adil Karjauv, and In So Kweon. Revisiting batch normalization for improving corruption robustness. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pp. 494-503, 2021.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. In Advances in Neural Information Processing Systems, 2011.
Jie Chang, Zhonghao Lan, Changmao Cheng, and Yichen Wei. Data uncertainty learning in face recognition. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2020.
Prithvijit Chattopadhyay, Yogesh Balaji, and Judy Hoffman. Learning to balance specificity and invariance for in and out of domain generalization. In European Conference in Computer Vision (ECCV), 2020.
Guangyao Chen, Peixi Peng, Li Ma, Jia Li, Lin Du, and Yonghong Tian. Amplitude-phane combination: Rethinking robustness of convolutional neural networks in frequency domain. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
Liang-Chied Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Patlern Analysis and Machine Intelligence, 40(4): 834-848, 2018.
Sungha Choi, Sanghun Jung, Huiwon Yun, Joanne T. Kim, Seungryong Kim, and Jaegul Choo. Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 11580-11590, 2021.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Yongxing Dai, Xiaotong Li, Jun Liu, Zekun Tong, and Ling- Yu Duan. Generalizable person re- identification with relevance-awee aware mixture of experts. Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 16145-16154, June 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, 2009.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, pp. 1126-1135. 2017.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In Proceedings The 33rd International Conference Machine Learning. volume 48, pp. 1050-1059, 2016.
Shang- Hua Gao, Qi Han, Duo Li, Ming- Ming-Ming Cheng, and Pai Peng. Representative batch normal- ization with feature calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8679, 2021a.
Shang-Hus Hua Gao, Qi Han, Duo Li, Pai Peng, Ming-Ming Cheng, and Pai Peng. Representative batch normalization with feature calibration. 2021b.
10

---

Published as a conference paper at ICLR 2022
Yixiao Ge, Dapeng Chen, and Hongsheng Li. Mutual mean-teaching: Pseudo label refinery for unsu- pervised domain adaptation on person re-identification. In International Conference on Learning Representations, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition (CVPR), 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. International Conference on Learning Representations, 2019.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple method to improve robustness and uncertainty under data shift. In International Conference on Learning Representations, 2020.
X. Huang and S. Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1510-1519, 2017.
Zeyi Huang, Haohan Wang, Eric Xing, and Dong Huang. Self-challenging improves cross-domane generalization. In The European Conference Computer Vision (ECCV), pp. 124-140, 2020.
Diederik Kingma and Max Welling. 1rXivv1312.611 14, 2013.
Auto-encoding variational
bayes.
arXiv preprint
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Boyi Li, Felix Wu, Ser-Nam Lim, Serge Belongie, and Kilian Q Weinberger. On feature normal- ization and data augmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 12383-12392, 2021.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning generalize: Meta-learning for domain generalization. Proceedings of the AAAI Conference on Artificial Intelligene, 2018a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. Domain generalization with adver- sarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5400-5409, 2018b.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), 3018c.
Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing do- main gap by reducing style bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8699-8699, 2021.
Oren Nuriel, Sagie Benaim, and Lior Wolf. Permuted adain: Reducing the bias towards global statistics in image classification. In Proceedings of the IEEE/CVF IEEF/CVF-Confereene Conference Computer Vision and Pattern Recognition (CVPR), pp. 9482-9491, 2021.
Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at once: Enhancing learning and generalization capacities via ibn-net. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Vihari Piratla, Praneeth Netrapalli, and Sunita sarawagi. Efficient domain generalization via common-specn speciffc low-rank decomposition. In Proceedings of the 37th International Conference Machine Learning, volume 119, pp. 7728-7738. 2020.

---

Published as a conference paper at ICLR 2022
Stephan R. Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European Conference on Computer Vision (ECCV), pp. 102-118, 2016.
Ergys Ristani, Francesco Solera, Roger S. Zou, R. Cucchiara, and Carlo Tomasi. Performance measures and data set for multi-target, multi-camera tracking. In ECCV Workshops, 2016.
Seonguk Seo, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. In The European Confer- ence on Computer Vision (ECCV), pp. 68 83, 2020.
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross- coos-ggradeet training. International Conference on Learning Reprexentations, 2018.
Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in gans. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2 1..iv:2108.132220 13224, 2021.
Yichun Shi and Anil K. Jain. Probabilistic face embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, lya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15556))1999-1958 2014.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008.
V. Vapnik. Principles of risk minimization for learning theory. In Advances in Neural Information Processing Systems, volume 4, 1992.
Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions neural net- works, 10(5):988-999) 1999.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 6438- 2019.
Haoran Wang. Tong Shen, Wei Zhang, Lingyu Duan, and Tao Mei. Classes matter: A fine- grained adversarial approach to cross cross-domain semantic segmentation. In The European Conference on Computer Vision (ECCV), 2020a.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Wenjun Zeng. and Tao Qin. Generalizing to unseen domains: A survey domain generalization. arXiv preprint a:X09.2103.03097, 2021.
Ximei Wang, Ying Jin, Mingsheng Long, Jianmin Wang, and Michael Jordan. Transferable nor- malization: Towards improving transferability of deep neural networks. In Advances in Neural Information Processing Systems, volume 32, 2019a.
Ximei Wang, Mingsheng Long, Jianmin Wang, and Michael Jordan. Transferable calibration with lower bias and variance in domain adaptation. In Advances in Neural Information Processing Systems, volume 33, pp. 19212-19223, 2020b.
Yulin Wang, Xuran Pan, Shiji Song, Hong Zhang, Gao Huang, and Cheng Wu. Implicit semantic data augmentation for deep networks. Advances Neural Information Processing Systems, pp. 2-614-12623, 2019b.
12

---

Published as a conference paper at ICLR 2022
Zhenlin Xu, Deyi Liu, Junlin Yang. Colin Raffel, and Mare Niethammer. Robust and generalizable visual representation learning via random convolutions. In International Conference on Learning Representations, 2021.
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6022 6031, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In International Conference Learning Representations, 2018.
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: A benchmark. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1116-1124, 2015.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In The European Conference on Computer Vision (ECCV), pp. 561-578, 2020a.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain adaptive ensemble learning. arXiv preprint arXiv:2003.07335 2020b.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. arXiv preprint 2rXiv:2103.0222) 2021a.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference Learning Representations, 2021b.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.I ALGORITHM
The algorithm of the proposed method is illustrated in Algorithm 1
Algorithm 1: The algorithm the proposed method (DSU)
Input:
Intermediate feature x RBxCxHX probability p to forward this module; Output: Intermediate feature 7 E R BBBCXXRW after considering potential statistics shifts; Sample po (O, 1);
if po and Training then
Compute the channel-wise mean and standard deviation of each instance in mini-batch;
a(x) HW
W
(² HW (Tb.c,h.w - p(z))²
Uncertainty estimation on feature statisties:
(x) Σ
(x) (z)))
h=1
(x) B
Σ (obc(z) (z b=1
Compute the synthetic feature statistics randomly sampling from the given Guassian distributions; 3(x) = p(z) + ep (x),
10
cu ~ N(,,
11
y(x) = (x) + c., E., (x),
e, 1.
12
Obtain the feature after considering potential statistics shifts;
13
z = 7()) z-p(z)
ov3) + B(x).
return the feature E with uncertain feature statistics.
else
adopt the original feature and skip this module.
end
A.2 MULTI-DOMAIN GENERALIZATION ON OFFICE HOME.
In addition to multi-domain classification experiments PACS, further evaluate the effective- ness the proposed method on Office-Home /Venkateswara et al. 2017). which contains 15,500 images of 65 classes for home and office recognition. The experiment results with ResNet18 back- bone are shown Table can be observed that our method brings obvious improvement over the baseline method and also outperforms the competing methods. By introducing the feature statistics uncertainty, the models trained with our method can learn to alleviate the domain perturbations, such the style information, and obtain more domain-invariant features. For example, huge improve- ment can be observed from the results on Clipart, which is a domain with much different style from others.
Table 7: Experiment results of Office-Home multi-domain classification task.
Method
Reference
Art
Clipart
Product
Real
Average (%)
Baseline
58.8
48.3
74.2
76.2
64.4
Mixup (Zhang et al.
2018
ICI.R 2018
49.3
74.7
76.1
64.6
CrossGrad (Shankar et al.
2018
ICLR 2018
58.4
49.4
73.9
75.8
64.4
Manifold Mixup Verma et al.
2019)
ICML 2019
56.2
46.3
73.6
75.2
62.8
CutMix Yun al.
2019
ICCV 2019
57.9
48.3
74.5
75.6
64.1
RSC (Huang a 2020
ECCV 2020
58.4
47.9
71.6
74.5
63.1
L2A-OT (Zhou et al.
2020g
ECCV 2020
60.6
50.1
74.8
77.0
65.6
MixStyle (Zoou et al
202Tb
ICLR 2021
58.7
53.4
74.2
75.9
65.5
DSU
Ours
60.2
54.8
74.1
75.1
66.1
14

---

Published as a conference paper at ICLR 2022
A.3 CHOICE OF UNCERTAINTY DISTRIBUTION
Table 8: Intensive study about different vanilla Gaussian distributions with pre-defined standard deviation.
Choice
Baseline
Rand(109)
Rand(10-1)
Rand(10-3)
Rand(10-3)
DSU
PACS
79.0
76.9
81.2
79.3
79.1
84.
GTAS to Cityscapes
37.0
38.2
39.8
40.1
38,9
43.1
Besides the analysis of the uncertainty estimation in the ablation study, we also conduct more in- tensive study about the effects of pre-defined uncertainty estimations. Specifically, Rand( denotes directly imposing random shifts draw from fixed Gaussian (0, The intensive study is shown in Table It can be observed that the results of different fixed distributions are all much lower than the proposed method. Some conclusions could be obtained from the results. (a): Imposing excessive uncertainty might harm the model training and degrade the performance. (b): The best fixed value of uncertainty estimation might vary from different tasks. By contrast, the proposed method can be adaptive to different tasks without any manual adjustment. Table 9: Study about the effects of sharing the same uncertain distribution among different channels.
Choice PACS
Baseline
Channel-share
DSU
79.0
80.2
84.
GTA5 to Cityscapes
37.0
39.3
43.1
We also conduct the experiment to test the effectiveness of treating different channels with different potentials. Channel-share denotes all channels of the sample share the same uncertainty distribution, i.e., using the average uncertainty estimation among channels. As shown in Table g the results in- dicate that sharing the same uncertain distribution among different channels is less effective, which ignores the different potentials of channels and will limit their performances. Meanwhile, the pro- posed method explicitly considers the different potentials of different channels and brings better performances.
A.4 T-SNE VISUALIZATION
To analyze the effects on feature representations, we vi- sualize the feature representation vectors of different cat- egories in unseen domain with t-SNE (Van der Maaten & Hinton 2008) Figure The features the same cate- gory become more compact benefiting from the proposed method. Because our method can alleviate the domain perturbations during training and make the model focus on content information, obtaining more invariant features representations.
Baseline
Ours
Figure 7: The t-SNE visualization on unseen PACS domain.
A.5 COMPARISONS TO THE RELATED METHODS
Some related methods (hhou et al.) 2021b Nuriel et al. 2021) also tackle the domain generalization problem by producing synthetic feature statistics. Specifically, we denote the random shuffle copies of the batch feature as ã (h) PAdaIN (Nuriel et al. (2021)) generate new samples by swapping feature statistics between the batch samples applied with random permutation, where 3(x) = (x))) and 7(x) = ( MixStyle (Zhou et al. (2021b)) generates synthesized domain samples by mixing feature statisties information two instances, where ((x) Ap(x:) + (1
(T) and 7(x) (x) + (1 (T) and À E (0, 1) is random interpolation weight.
Despite the success, they typically adopt linear manipulation on pairwise samples to generate new feature statistics, which limits the diversity of synthetic changes. Specifically, the direction of the variants is determined by the chosen reference sample and the internal operation also restricts the variant intensity. Our method, not relies on specific reference sample, is based on the Gaussian distribution that can produce not only linear changes but diverse variants with more possibilities. Due to the boundless range of the Gaussian distribution, feature statistics beyond the scope of training domain, which also breaks the limitation of inner interpolation between training samples. The visualization of the comparisons is shown in Figure g
method has the ability to generate
15

---

Published as a conference paper at ICLR 2022
reference sample
anchor sample
samples
synthetic samples
anchor sample
source domain
*
probabilistic
representation
[a]: visualization results linear manipulation
visualization method
Figure 8: Comparisons with related methods. The variants produced by previous pairwise-bare based methods are restricted by the combination of chosen sample pair, while our method can generate various feature statistics variants with different combination of directions and intensities.
A.6 WITHIN-DATASER PERFORMANCE
In Table we tested the tithin-dataset performance on the large-scale dataset ImageNet, denoted as "Clean" We observed that the top-1 error rate declines from 23.8% 23.4% after training with the proposed DSU, indicating that DSU does not sacrifice the in-domain performance gain the benefits on out-of-distribution domains. The reason might be that instances the testing set may not always fall into exactly the same distribution of the training set, and they still have slight statistic shifts (Gao et al., 2021a). The proposed DSU can help the trained model improve the robustness to statistics shifts and thus gain better performance in within-dataset ImageNet.
Besides the experiments on ImageNet, we also supplement the within-dataset performances on PACS. According to the multi-source training protocol PACS (Li et al. (017)) the within-domain performance is averaged over multiple training-doman domain datasets (P.A, denotes Art, Cartoon, Photo and Sketch respectively). As shown in Table T0, it be observed that the within-dataset performance of our DSU also slightly beats the baseline, verifying the conclusion as on ImageNet.
Table 10:
performance on PACS. P,A,C,S denote Photo, Art Painting, Cartoon, and
Sketch respectively.
Method Baseline
Reference
P.C.S
P.A.S
A,C.S
P.A.C
Average (%)
95.70
95.28
94.22
96.58
95.44
DSU
Ours
96.20
96.32
95.17
97.20
96.21
A.7 ABLATION STUDY ON BATCH SIZE
In Table we conduct an ablation study on the batch size. As shown in the table, consistent performance gains are observed with various batch sizes on PACS. Note we use the batch size of 64 in our paper, following the original setting in PACS (Li et al. 2017) for fair comparison.
Table 11: Ablation study on the effects of batch size.
batchsize
Reference
16
32
64
128
256
Baseline
81.0
80.2
79.0
77.8
75.6
DSU
Ours
84.9 (+3.9)
84.5 (+4.3)
84.1 (+5.1)
82.1 (+4.3)
80.4 (+4.8)
16


Reviews:
[{"review_id": "SwlKCOkpG4r", "reviewer": "Reviewer_jq8e", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n(1) The proposed method can capture the uncertainty of the feature domain, which is very important to address the domain shift issue.\n\n(2) The proposed method can achieve some improvement on cross-dataset evaluation for multiple vision tasks compared with other domain generalization methods.\n\n(3) This paper is well written and well-organized. \n\nWeaknesses:\n\nFor the method:\n\n(1) The proposed method is simple and is similar to batch normalization. Can traditional batch normalization method also improve model generalization ability? Is the proposed method significantly better than traditional batch normalization?\n\n(2) Generating the class-preserving augmented features is important since we do not want the generated features to be far away from the original class domain. The proposed feature augmentation method did not consider the class-preserving issue, i.e., how to guarantee the augmented feature is still located within the feature domain of the same class. \n\n(3) We may expect more discussion on why the domain generalization can be improved by modeling the probabilistic distribution of features. Is there a specific theory or reason? \n\nFor the experiments:\n\n(1) The authors can provide some within-dataset experiments for comparison.\n\n(2) According to Eq.(3) and Eq.(4), the variance of feature statistics is related to batches, it is better to perform an ablation study of the influence of batch size.\n\n(3) For the image classification problem, the performance of DSU drops on Photo style compared with the baseline. Are there some explanations?\n\n", "summary_of_the_paper": "This paper proposed a feature augmentation method for domain generalization by generating probabilistic feature statistics, where the distribution of feature statistics enlarges the feature domain with possible domain shifts. The proposed method is evaluated on various vision tasks, including image classification, semantic segmentation, and instance retrieval. It achieves competitive results compared with recent domain generalization methods. ", "summary_of_the_review": "Overall, this paper is well-written and well-organized for both theories and experiments. The idea is interesting and is easy to implement. The experiment results are empirically sound. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "M1ifEAhZz4b", "reviewer": "Reviewer_SDxo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Overall, the paper is well written and nicely organized.\n\n2. The proposed method is intuitive and simple. It adds randomness towards the feature statistics to handle out-of-distribution testing. The randomness is also drawn from the observed uncertainty estimations from data, with the hope that the uncertainty estimations from mini-batch could provide variation range to cover the possible unknown domain shifts. The method is simple yet very effective according to the experimental results. This is a big plus for the method.\n\n3. The method is tested on various benchmarks, and shows strong performance compared to recent state-of-the-art methods.\n\n4. Some parts of the paper feel repetitive. For example, Section 3.1, related work and introduction have similar contents, which could be reorganized to improve the presentation and writing.", "summary_of_the_paper": "This paper studies the problem of out-of-distribution generalization by modeling uncertainty in feature statistics. It improves the network generalization ability by modeling the uncertainty of domain shifts with synthesized feature statistics during training. Instead of being deterministic values, the feature statistics are hypothesized to follow a multivariate Gaussian distribution. The proposed method is tested on various tasks including image classification, semantic segmentation, and instance retrieval, and shows strong performance compared to state-of-the-art methods.", "summary_of_the_review": "In summary, the method is simple yet effective, showing strong performance on various benchmarks.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "KS9u3nVh0GF", "reviewer": "Reviewer_xdeV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The proposed multivariate Gaussian based approach makes sense and technically sound to me. \n2. Extensive experiments, ablation study, and sufficient analysis validated the approach empirically.\n3. Writing is good and easy to follow.\n\nWeakness:\n1. The proposed approach seems to be a small change to AdaIN. \n2. Instead of sampling from a Gaussian distribution, can we just normalize it to be an uniform destruction? Similar idea to Batch Normalization, normalizing the distribution to a standard one seems to be an easier choice.", "summary_of_the_paper": "In this paper, the authors proposed to model uncertainty with multivariate Gaussian distribution for better network generalization. Experiments on multiple benchmark datasets show improved result. Several visualizations and analysis also illustrate the effectiveness of the proposed approach.", "summary_of_the_review": "Based on the above analysis, I am slightly leaning towards acceptance for now.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers the important problem of performance degradation under distribution shift and proposes a simple yet effective method to alleviate this problem. They do so by considering feature statistic to be non-deterministic and rather a multivariate Gaussian distribution.  The model can be integrated into networks without additional parameters and experiments show that it works better than BN as well as if the assumed distribution was uniform. The latter was added during rebuttal period.

There were two main concerns regarding distinguishing the work from AdaIN and baseline that were addressed during rebuttal and some parts of the paper were re-written to address repetition.
Decision: Accept (Poster)