SelfNorm and CrossNorm for Out-of-Distribution Robustness
Authors: Zhiqiang Tang
Yunhe Gao
Yi Zhu
Zhi Zhang
Mu Li
Dimitris N. Metaxas
Venue:  (2021)

Abstract: Normalization techniques are crucial in stabilizing and accelerating the training of deep neural networks. However, they are mainly designed for the independent and identically distributed (IID) data, not satisfying many real-world out-of-distribution (OOD) situations. Unlike most previous works, this paper presents two normalization methods, SelfNorm and CrossNorm, to promote OOD generalization. SelfNorm uses attention to recalibrate statistics (channel-wise mean and variance), while CrossNorm exchanges the statistics between feature maps. SelfNorm and CrossNorm can complement each other in OOD generalization, though exploring different directions in statistics usage. Extensive experiments on different domains (vision and language), tasks (classification and segmentation), and settings (supervised and semi-supervised) show their effectiveness.

Under review as a conference paper at ICLR 2021
SELFNORM AND CROSSNORM DISTRIBUTION ROBUSTNESS
FOR OUT-OF-
Anonymous authors Paper under double-blind review
ABSTRACT
Normalization techniques are crucial in stabilizing and accelerating the training of deep neural networks. However, they are mainly designed for the indepen- dent and identically distributed (IID) data, not satisfying many real-world out-of- distribution (OOD) situations. Unlike most previous works, this paper presents two normalization methods, SelfNorm and CrossNorm, to promote OOD general- ization. SelfNorm uses attention recalibrate statistics (channel-wise mean and variance), while CrossNorm exchanges the statisties between feature maps. Self- Norm and CrossNorm can complement each other in OOD generalization, though exploring different directions in statistics usage. Extensive experiments differ- ent domains (vision and language), tasks (classification and segmentation), and settings (supervised and semi-supervised) show their effectiveness.
INTRODUCTION
Normalization methods, c.g., Batch Normalization (loffe & Szegedy 2015). Layer Normalization (Ba et al. 2016), and Instance Normalization (Ulyanov al., (2166), play pivotal role in training deep neural networks. Most of them try make training more stable and convergence faster, as- suming that training and test data come from the same distribution. However, few studies investigate normalization in improving OOD generalization in real-world scenarios. For example, image cor- ruptions (Hendrycks & Dietterich 2019), e.g., snow and blur, can cause test data out the clean training distribution. Moreover, training synthetic data (Richter et al., 2016) to generalize to realistic data can significantly reduce the annotation burden. This work aims to encourage the in- teraction between normalization and OOD generalization. Specifically, we manipulate feature mean and variance to make models generalize better to out-of-distribution data.
Our inspiration comes from the observation that channel-wise mean and variance of feature maps carry some style information. For instance, exchanging the RGB means and variances between two instances can transfer style between them, shown in Figure (a). For many tasks such as CIFAR classification (Krizhevsky et al. 2009), the style encoded by channel-wise mean and variance is usually less critical in recognizing the object than other information such as object shape. Therefore, we propose CrossNorm that swaps the channel- wise mean and variance of feature maps. CrossNorm can augment styles in training, making the model more robust to appearance changes.
Furthermore, given one image in different styles, we can reduce their style discrepancy if adjusting their RGB means and variances properly, as illustrated Figure (b). Intuitively, the style recali- bration can reduce appearance variance, which may be useful in bridging distribution gaps between training and unforeseen testing data. To this end, we propose SelfNorm to use attention (Hu et al.) 2018) to adjust channel-wise mean and variance automatically.
interesting to analyze the distinction and connection between CrossNorm and SelfNorm. At first glance, they take opposite actions (style augmentation the same tool: channel-wise statistics and pursue the same goal: OOD robustness. Additionally, CrossNorm increase the capacity of SelfNorm by style augmentation. SelfNorm, with the help from CrossNorm, can generalize better to OOD data.
style reduction). Even so. they use
Concept and Intuition. The style concept here refers family of weak cues associated with the semantic content of interest. For instance, the image style in object recognition can include many appearance-related factors such color, contrast, and brightness. Style sometimes may help in

---

Under review as a conference paper at ICLR 2021
(a) Switch RGB mean and variance
(b) Recalibrate RGB mean and variance
Figure 1: CIFAR examples of exchanging (Left) and adjusting (Right) RGB mean and variance.
decision-making, but the model should weigh more more vital content cues to become robust. To reduce bias rather than discard it, we use CrossNorm with probability in training. The insight beneath CrossNorm is that each instance, or feature map, has its unique style. Further, style cues are not equally important. For example, the yellow color seems more useful than other style cues in recognizing orange. In light of this, the intuition behind SelfNorm is that attention may help emphasize essential styles and suppress trivial ones.
Assumption. Although
the channel-wise mean and variance to modify styles, do not assume that they are sufficient to represent all style cues. Better style representations are available with complex statistics (Li et al. (017) or even style transfer models (Ulyanov et al. 2017 Huang& & Belongie| 2017). We choose the first and second-orde statistics mainly because they are simple, efficient to compute, and can connect normalization to out-of-distribution generalization. In summary, the key contributions are:
We propose SelfNorm and CrossNorm, two simple yet effective normalization techniques to enhance ootoo--distribution generalization.
SelfNorm and CrossNorm form unity of opposites in using feature mean and variance for model robustness.
They are domain agnostic and can advance state-of-the-art robustness performance for dif- ferent domains (vision or language), settings (fully or semi-supervised), and tasks (classi- fication and segmentation).
2 RELATED WORK
-ut-of-distributton generalization. Although the current deep models continue break records on benchmark IID datasets, they still struggle to generalize to OOD data caused by common corruptions (Hendrycks Dietterich 2019) and dataset gaps (Richter al. 2016). To improve the robustness against corruption, Stylize--ImageNe (Geirhos et al. (019) conducts style augmentation to reduce the texture bias of CNNs. Compared to it, CrossNorm has two main advantages. First, CrossNorm is efficient as it transfer styles directly in the feature space of the target CNNs. However, Stylized- ImageNet relies on external style datasets and pre-trained style transfer models. Second, CrossNorm can advance the performance on both clean and corrupted data, while Lerliieed-ImaeeNee hurts clean generalization. contrast to the consistent styles within one dataset, the external ones can result in massive distribution shifts. Recently, AugMix (Hendrycks al., 2020cc trains robust models by mixing multiple augmented images based on random image primitives or image-to-image networks (Hendrycks al. (2020a). Adversarial noises training (ANT) (Rusak al. (2020) can also improve the robustness against corruption. CrossNorm is domain agnostic and orthogonal to AugMix and ANT, making it possible for their joint application. Moreover, unsupervised domain adaptation is also useful for corruption robustness in some situations (Schneider et al. 2020).
Besides common corruptions. generalization with distribution gaps Riccheer et al., 2016) across different datasets also suffers from problems. IBN (Pan et al. 2018) mixes instance and batch normalization to shrink the domain distances. SelfNorm can bridge the domain gaps by style recali- bration. Domain randomization (uue et al.) 2019) uses style augmentation for domain generalization on segmentation datasets. It suffers from the same issues of Syylized-ImageNet as it also uses pre- trained style transfer models and additional style datasets. By contrast, CrossNorm is more efficient and balances better between the source and target domains' performance. Beyond the vision field, many natural language processing (NLP) applications also face the dut-of-distributtin generaliza- tion challenges (Hendrycks et al. 2020b). Benefiting from the domain domain-agnostt property, SelfNorm and Noom also improve model robustness in the NI.P area.
2

---

Under review as a conference paper at ICLR 2021
Standardized
A A-44
Statistics recalibration
[ = (ug.aa2l ''A onon
+ 4B
(MA.. ox)
(Mi op.
A HA
8 Ma
Standardized
B Hu
Standardized
SelfNorm
(b) CrossNorm
Figure 2: SelfNorm (left) and CrossNorm (right). SelfNorm uses attention to recalibrate the mean and variance of feature map, while CrossNorm swaps the statisties between pair of feature maps.
No
Input
Training?
CrossNorm
SelfNorm
Output
Figure 3:
Flowchart for SelfNorm and CrossNorm. SelfNorm learns in training but functions in lesting, while CrossNorm works in training.
Normalization and attention. Batch Normalization (loffe Szegedy 2013) is a milestone tech- nique that inspires many following normalization methods such Instance Normalization (Ulyanov) (ct al. 2016). Layer Normalization (Ba et al.) 2016). and Group Normalization (Wu & He (018)) Recently, some works integrate attention (Hu et al. 2018) into feature normalization. Mode nor- malization (Deecke et al. 2018) and attentive normalization (Li et al. 2019) use attention to weigh mixture of batch normaliza lizations. Examplar normalization (Zhang et al. (0200) learns to combine multi-type normalizations by attention. By contrast, SelfNorm uses attention with only instance normalization. More importantly, different from previous normalization approaches, SelfNorm and CrossNorm are to improve out-of-disse di-sttiiibution generalization. In addition, SelfNorm is different from SE (Hu et al. 2018). though they use similar attention. First, SelfNorm learns to recalibrate channel- wise mean and variance instead of channel features in SE. Second, SE models the interdependency between channels, while SelfNorm deals with each channel independently. Also, SelfNorm unit, with (n), is more lightweight than SE one, of Onn²), where n denotes the channel number. The difference analysis here also applies to the Channel Attention (Zhang et al. 2018), similar to SE.
Data augmentation. Data augmentation is an important tool in training deep models. Current pop- ular data augmentation techniques are either bebel-preserving (Cubuk et al.] 2019a Lim al. 2019, Ho et al. 2019) tabel-perturbing (Zhang et al.) 2017, Yun al.) 2019). The label-preserving methods usually rely on domain-specific image primitives, c.g., rotation and color, making them inflexible for tasks beyond the vision domain. The label-perturbing techniques mainly work for classification and may have trouble in broader applications, e.g., segmentation. CrossNorm, as a data augmentation method, is readily applicable to diverse domains (vision and language) and tasks (classification and segmentation). The goal of CrossNorm is to boost out-of out-of-distribution generaliza- tion, which is also different from many former data augmentation methods.
SELFNORM AND CROSSNORM
Background. Technically, SelfNorm and CrossNorm share the origin: instance normalization (Ulyanov et al.) 2016). In 2D CNNs, each instance has C' feature maps of size H x W. Given A E g instance normalization first normalizes the feature map and then conducts affine
transformation:
A
MA
+ 3,
σ
where μA and a the mean and standard deviation; and 3 denotes learnable affine parameters. As shown in Figure| T and also pointed out by the style transfer practices (Dumoulin et al. 2016) Ulyanov al. 2017 Huang & Belongie 2017), 7) nA and a A encode some style information.

---

Under review as a conference paper at ICLR 2021
SelfNorm. SelfNorm replaces B and with recalibrated mean μ'^ (( and standard deviation d'A = g(AA. AA A. as illustrated in Figure 2 where f and are the attention functions. The adjusted channel becomes:
1A-HA V'
A
+ 1/A
(2)
σA
As f and g learn to scale μ사 and O based on themselves, A normalizes itself by gelf-gating, hence SelfNorm. SelfNorm is inspired by the fact that attention can help the model emphasize informative features and suppress less useful ones. In terms of recalibrating μA and σ SelfNorm expects to highlight the discriminative styles and understate trivial ones. In practice, (FC) network to wrap attention functions f and g. The architecture efficient as its input and output are both scalars. Since each channel has its independent statistics, SelfNorm recalibrates each channel separately using C' lightweight FC networks, hence the complexity of O(C).
fully connected
CrossNorm. CrossNorm exchanges μ and a of channel A with μ and o of channel B, i.e. changing 3 and y to each other's 's and σ, shown in Figure 2
MA
B
B PB
O
+ PB
OA
PAA
(3)
OA
where A and B seem normalize each other, hence CrossNorm. CrossNorm is motivated by the key observation that a target dataset, such as a classification dataset, has rich, though subtle, styles. Specifically, each instance, or even every channel, has its unique style. Exchanging the statistics can perform efficient style augmentation, reducing the style bias in gocision-making In mini-batch training, we turn on CrossNorm with some probability.
Unity of Opposites. SelfNorm and CrossNorm both start from instance normalization but head in opposite directions. SelfNorm recalibrates statistics focus on only necessary styles, reducing standardized features (zero-mean and unit-variance) and statistics mixtures' diversity. In contrast, CrossNorm transfers statistics between channels, enriching the combinations of standardized fea- tures and statistics. They perform opposite operations mainly because they target at different stages. SelfNorm dedicates to style recalibration during testing. while CrossNorm functions only in train- ing. Note that SelfNorm is a learnable module, requiring training to work. Figure 3 shows the flowchart of SelfNorm and CrossNorm. Additionally, SelfNorm helps make the model less sensitive to appearance changes, while CrossNorm aims to lessen the model's style bias. Despite these dif- ferences, they both can facilitate out-of ontoof-distribution generalization. Further, CrossNorm can boost the performance of SelfNorm because its style augmentation can prevent SelfNorm from overfitting to specific styles. Overall, the two seemingly opposed methods form a unity of using normalization statistics advance -uttt--distriuttion robustness.
3.1 CROSSNORM VARIANTS
The core idea of CrossNorm is to swap mean and variance between channel features. For 2D CNNs, given one instance X, E RC*H H xW CrossNorm can exchange statisties between its channels:
{(A, E (X.... Xaa.) | j.0 i,j C}.
(4)
where A and B refer to the channel pair in Equation 3 two instances X, Y E RCxH*W given, CrossNorm swap statistics between their corresponding channels, i.e., A and B become:
((A, B) (X...) Yr.)) < <i<C)-
Compared with one-instance CrossNorm, the two-instance tends to consider instance-level style instead of channel-level.
Moreover, distinct spatial regions probably have different mean and variance statisties. To promote the style diversity, we propose to crop regions for CrossNorm:
{(A, B) E (crop(A), crop(B)) Ferop 2 t
(6)
where the crop function returns a square with area ratio r no less than a threshold t(0 < <ts 1). The whole channel is special case in cropping. There are three cropping choices: content only, style only, and both. For content cropping, we crop A only when we use its standardized feature map. In other words,
cropping applies to A when it provides its statistics to B. Cropping both means cropping A and B no matter we employ their standardized feature map or statistics. The cropping strategy can produce diverse styles for both the two-instance and one-instance Norms.
4

---

Under review as a conference paper at ICLR 2021
3.2 MODULAR DESIGN
SelfNorm and CrossNorm can naturally work in the feature space, making it flexible to plug them into many network locations. Two questions come: how many units are necessary and where to place them? To simplify the questions,
turn the modular design by embedding them into network cell. For example, in ResNet (He et al. 2016), we put them into a residual module. The search space significantly shrinks for the limited positions choices in experiments. The modular design allows using multiple SelfNorms and CrossNorms in network. We will show in the ablation study that accumulated style recalibrations are helpful for model robustness. Since excessive style augmentations are harmful (Geirhos et al.) (201)) we ran- domly turn on only some CrossNorms in a forward process. Random sampling encourages diverse augmentations even though the same data pass multiple times.
residual module. We will investigate the position
4 EXPERIMENT
We evaluate SelfNorm and CrossNorm on out-of-distribttion data that arise from image corruptions and dataset differences. The evaluation not only supervised and semi-supervised settings but also image classification and segmentation tasks. In addition to the vision tasks, we also apply them NLP task. Due to limited space, we leave all ablation studies in the appendix.
Image classification datasets. We use benchmark datasets: CIFAR-10 (Krizhevsky et al. 2009)
CIFAR-100, and ImageNer(Deng al.) 2009). To evaluate the model robustness against corruption, we use the datasets: CIFAR- 10-C, CIFAR-I0O-C, and ImageNet-C (Hendrycks & Dietterich] 2019). These datasets are the original test data poisoned by 15 everyday image corruptions from 4 general types: noise, blur, weather, and digital. Each noise has 5 intensity levels when injected into images.
Image segmentation datasets. We further validate our method using domain generalization set- ting. where the models are trained without any target domain data and tested on the unseen domain. We use the synthetic dataset Grand Theft Auto V (GTA5) (Richter et al., (016)) as the source domain and generalize to the real-world dataset Cityscapes (Cordts et al. 2016). GTA5 has the training, vali- dation, and test divisions of 12,403, 6,382, and 6, 181, more than those of 2,975, 500, and 1,525 from Cityscapes. Despite the differences, their pixel categories are compatible with each other, making it possible to evaluate models' generalization capability from one to another.
Sentiment classification datasets. Besides vision tasks, we demonstrate that our method can also work well on NLP tasks. We use the out-of-distribution (OOD) generalization setting in binary sentiment classification. The model is trained on IMDb dataset (Maas et al., 2011) and is tested on SST-2 testing dataset (Socher et al., 2013). The IMDb dataset collects highly polarized full-length lay movie reviews with 25,000 positive and 25, 000 negative reviews. The SST-2 contains 9613 and 1821 reviews for training and testing, which is also a binary sentiment classification dataset but instead contains pithy expert movie reviews.
For image classification, we use test errors to measure the robustness. Given corruption type and severity s, let E denote the test error. For CIFAR datasets, we use the average over 15 corrup- tions and 5 severities: 1/75 E15 >5
Metric.
s==1 Fec.. contrast, for ImageNet, we normalize the corrup- tion errors by those of AlexNet (Krizhevsky et al. 2012): 1/15 Σ15 - E E c,s Net) The above two metrics follow the convention (Hendrycks al. 2020c) and are denoted as mean corruption errors (mCF) whether they are normalized or not. Different from classification, segmen- tation use the mean Intersection over Union (mIoU) over all categories as metric. For sentiment classification, we report accuracy the metric.
Hyper-parrameter. In the experiments, a SelfNorm unit uses one fully connected layer, followed by Batch Norm and sigmoid layer. We put CrossNorm ahead of SelfNorm, and plug them into every cell in a network, e.g., each residual module in a ResNet. During training, we turn on only some CrossNorms with probability to avoid excessive data augmentation. Usually, we conduct grid search on four combinations of active numbers (1, 2) and probability (0.25, O.5). For CrossNorm with cropping, we sample the bounding box ratio uniformly and set the threshold = 0.1.
5

---

Under review as a conference paper at ICLR 2021
Table 1: mCE (%) comparison of CIFAR-IO-C and CIFAR-100-C. SelNNorm&CrossNom (SNCN) obtains lower errors than most previous methods with different backbones. Albeit some higher errors than AugMix, it is totally domain agnostic without relying on the image primitives, e.g., rotation, in AugMix. As SNCN and AugMix are orthogonal, their joint usage brings new state-of-the-ar results.
CIFAR-IO-C
Basic
Cutout
Mixup
CutMix
AutoAug
AdvTr.
AugMix
SNCN
SNCN+AugMix
AllConvNet
30.8
32.9
24.6
31.3
29.2
28.1
15.0
17.2
11.8
DenseNet
30.7
32.1
24.6
33.5
26.6
27.6
12.7
18.5
10.4
WideResNet
26.9
26.8
22.3
27.1
23.9
26.2
11.2
16.9
9.9
ResNeXt
27.5
28.9
22.6
29.5
24.2
27.0
10.9
15.7
9.1
Mean
29.0
30.2
23.5
30.3
26.0
27.2
12.5
17.0
10.3
CIFAR-I00-C
Basic
Cutout
Mixup
CutMix
AutoAug
AdvTr.
AugMix
SNCN
SNCN+AugMix
AllConvNet
56.4
56.8
53.4
56.0
55.1
56.0
42.7
42.8
36.8
DenseNet
59.3
59.6
55.4
59.2
53.9
55.2
39.6
48.5
37.0
WideResNet
53.3
53.5
50.4
52.9
49.6
55.1
35.9
43.7
33.4
ResNeXt
53.4
54.6
51.4
54.1
51.3
54.4
34.9
40.8
30.8
Mean
55.6
56.1
52.6
55.5
52.5
55.2
38.3
43.5
34.7
Table 2: Clean error and mCE (%) of ResNe150 trained 90 epochs on ImageNet. SNCN, using simple domain-agnone agnostic statistics, achieves comparable performance as AugMix. Jointly applying SNCN with AugMix and IBN can produce the lowest clean and corruption errors.
Noise
Blur
Weather
Digital
Aug.
Cleam
Gauss.
Shol
Impuulse
Defocus
Glass
Motion
Zoom
Snow
Frost
Fog Bright Contrasi
Elastic Pixel JPEG
mCC
Standard
23,9
82
90
80
86
81
15
91
Patch Uniform
24.5
67
68
70
74
83
81
77
80
74
75
62
77
84
71
71
74.3
Random AA"
23.6
72
80
86
82
81
77
72
75
88
76.1
MaxBllr pool
23.0
74
76
74
86
78
77
77
72
63
68
86
71
73.4
SIN
27.2
70
77
84
76
82
74
75
69
69
80
77
73.3
AugMix
23.4
66
69
80
65
72
72
66
63
78
71
68.4
SNCN
23.3
66
67
65
77
89
76
80
72
72
67
59
47
83
62
72
70.4
ANgNt-
22.3
70
77
4.1
ROBUSTNESS AGAINST UNSEEN CORRUPTIONS FOR IMAGE CLASSIFICATION
Supervised training on CIFAR. Following AugMix (Hendrycks et al. 2020c). we evaluate Self-
Norm and CrossNorm with four different backbones: All Convolutional Network (Springenberg ct al. al. 2014), a DenseNet-BC (k= 12, d= 100) (Huang et 2017), a 40-2 Wide ResNet (Zagoruyko
& Komodakis 20166, and ResNeXt-29 (32x4) (Xie al. 2017). parameters in the AugMix Github repository
We also the same hyper-
According to Table TT SelfNorm and CrossNorm decrease the mean error by 122% on both CIFAR-10-C and CIFAR-100-C, outperforming most previous approaches on robustness against unseen corruptions. One possible explanation is that the corruptions, as demonstrated in Figure 12 mainly change image textures. SelfNorm and CrossNorm, through style recalibration and aug- mentation, may help reduce the texture sensitivity and bias, making the classifiers more robust to unseen corruptions. Also, the domain-agnosti SelfNorm and CrossNorm are orthogonal to Aug- Mix, which relies on domain-specific operations. Their joint application can continue to lower the mCEs by 2.2% and 3.6% on top of AugMix.
Supervised training on ImageNet. Following the AugMix Github repository, we train a ResNet-50 for 90 epochs with weight decay le-4. The learning rate starts from 0.1, divided by 10 at epochs 30 and 60. Note that AugMix reports the results of 180 epochs in their paper. For fair comparison, also train it 90 epochs in our experiments. Besides, we also add Instance-batch normalization (IBN) (Pan et al., 2018) in the final combination with AugMix. was initially designed for domain generalization but also boost model robustness against corruption.
Table gives the results on ImageNet. We can find that SelfNorm and CrossNorm can make the corruption eиror drop by 10.2%. The clean error also descends by 0.6% simultancously. More- over, applying SelfNorm and CrossNorm on top of AugMix can significantly lower its clean and corruption errors by 1.1 11% and 5.6%, achieving state-of-the-art performance. IBN also makes some contributions here since it is complementary to other components.

---

Under review as a conference paper at ICLR 2021
250 labels
1000 labels
FixMatch (WA)
24
FixMatch (RA)
60
6/4
22
FixMatch (RA)
mCE (%)
FixMatch (WA)
55%
3.6
CrussNorm
K1
16-
0.9
CrossNorm
3.5
14.
CrossNorm
30
CrossNorm
12
18
50
52
54
56
58
10
11
Clean error (%)
Clean ervor (%)
Figure 4: CrossNorm for semi- supervieee CIFAR-10 classification. We apply CrossNorm on top of FixMatch with weak augmentation (WA) (Left), or strong RandAugment (RA) (Right), For either case, CrossNorm (CN) can substantially reduce both clean and corruption errors. Compared with RA, CrossNorm performs domain agnostic data augmentation, easily applicable to new domains.
Semi-supervised training on CIFAR. Apart from supervised training, we also evaluate CrossNorm in semi-supervised learning. Following state-of-the-art FixMatch (Sohn et al.] (020) setting, we train a 28-2 Wide ResNet for 1024 epochs on CIFAR-10. The SGD optimizer applies with Nesterov momentum 0.9, learning rate 0.03, and weight decay 5e-4. The probability threshold to generate pseudo-labels is 0.95, and the weight for unlabeled data loss is 1. We sample 250 and 4,000 la- beled data with random seed 1, leaving the rest as unlabeled data. In each experiment, we apply CrossNorm to all data or only the unlabeled and choose the better one. Our experiments use the Py- which has higher errors than the FixMatch reported. We choose it because it has the most stars among all the Pytorch implementations on Github.
torch Fitca implementation? [
Figure showss the semi-supervised results. We run Fix.aatch with the strong RandAugment (Cubuk al., 2019b) or only weak random flip and crop augmentations. With either FixMatch version, CrossNorm can always decrease both the clean and corruption errors, demonstrating its effective- ness in sumi-supe training. Especially, with the help of CrossNorm, training with 250 labels even has 3% lower corruption eтror than with 1000 labels, according to the right sub-figure. Addi- tionally, two points are noteworthy here. First, we try FixMatch with only weak augmentations to simulate more general situations. For new domains other than natural images, humans may have the limited domain knowledge to design advanced augmentation operations. Fortunately, CrossNorm domain-agnose and easily applicable to such situations. Moreover, previous semi-supervised meth- ods mainly focus on in-distribution generalization. Here we introduce out-of--istributiio robustness as another metric for more comprehensive evaluation.
4.2 GENERALIZATION FROM SYNTHETIC TO REALISTIC DATA FOR IMAGE SEGMENTATION
Training setup. We perform domain generalization from GTA5 (synthetic) to Cityscapes (realistic), following the setting of IBN (Pan et al. 2018). It uses 1/4 training data in GTA5 to match the data scale of Cityscapes. We train the FCN (Long et al. 2015) with ResNet50 backbone in source domain GTA5 for 80 epochs with batch size 16. The network is initialized with ImageNet pre- trained weights. Then we test the model both the source domain and target domain. The training uses random scaling, flip, rotation, and cropping (713 x 713) for data augmentation. We use the 2-instance CrossNorm with style cropping in this setting. Besides, we re-implement the domain randomization (Yue et al., 2019) and make the training iterations the same as ours. It transfers the synthetic images to 15 auxiliary domains with ImageNet image styles.
Results. Based on Table 3, SelfNorm and CrossNorm both can substantially increase the segmenta- tion accuracy on the target domain by 8.5% and 10.6%. SelfNorm learns to highlight the discrimina- tive styles that are likely to share across domains. CrossNorm performs style augmentation to make the model focus more on domain-invariant features. SelfNorm and CrossNorm get comparable gen- eralization performance as state-of-ther thee--rrt IBN (Pan et al., 2018) and domain randomization (Yue) (et al., 2019). However, CrossNorm significantly outperforms the domain randomization method by 12.2% on the source accuracy. Because the domain randomization transfers external styles to the source training data, causing dramatic distribution shifts. Moreover, combining SelfNorm and CrossNorm gives the best generalization performance while still maintaining high source accuracy.
7

---

Under review as a conference paper at ICLR 2021
Table Segmentation results (mloU) on GTAV-Cityscapes domain generalization using FCN with ResNet50. SelfNorm (SN) and CrossNorm (CN) are comparable with IBN and domain randomiza- tion (DR) on the target domain. Combining SN and CN can achieve ttate-of-the-ar performance.
Methods
Baseline
IBN
DR
SN
CN
SNCN
Source
63.7
64.2
49.0
64.6
61.2
63.5
Target
21.4
29.6
32.7
29.9
32.0
36.5
Table 4: Accuracy (Acc) on OOD generalization for sentiment classification using GloVe embedding and ConvNets model. We train the model on IMDb source dataset and test on SST-2 target dataset.
Methods
Baseline
SN
CN
SNCN
Source 85.67
86.30 (f 0.63)
85.14 (L 0.53)
85.92 (T 0.25)
Target
71.86
73.93 (1 (.07)
73.03 (f 1.17)
74.91 (1 3.05)
4.3 DUT-OR-DISTRIBUTION GENERALIZATION FOR SENTIMENT CLASSIFICATION
Setup. We also conduct out-of distrriiibttinn generalization on the binary sentiment classification task in the NLP field to validate the versatility of SelfNorm and CrossNorm. The model is trained on the IMDb dataset and then tested on SST-2 dataset. Follow the setting of Henndrycks al. (020b), we use GloVe (Pennington et al. 2014) word embedding and the Convolutional Neural Networks (ConvNets) (Kim (014)) as the classification model. We use the implementation of ConvNets in this gram features within the review texts. The SelfNorm and CrossNorm are placed between the embedding layer and the convolutional layers. We use the Adam optimizer and train the model for 20 epochs.
repository The convolutional layers with three kernel sizes (3,4,5) are used to extract
Results. From Table 4 we can find that SelfNorm improves the performance in both the source and target domains by 2.07% and 0.63%. CrossNorm can also increase target accuracy without much degradation in the source domain. Combining them gives a 3.05% accuracy boost. This experiment indicates that SelfNorm and CrossNorm also work in the NLP area, not limited to the vision tasks. Despite the lack of intuitive explanations as for the image data, the mean and variance statistics in NLP data are also useful in facilitating out-of-distribution generalization.
4.4 VISUALIZATION
Apart from the quantitative comparisons, we also provide some visualization results of SelfNorm and CrossNorm to understand their effects better. is nontrivial to visualize them directly in feature space. To deal with this, we map the feature changes made by SelfNorm and CrossNorm back to image space by inverting the feature representations (Mahendran & Vedaldi 2015). For detailed experimental settings, refer to the appendix.
To visualize SelfNorm at network location, we first forward an image obtain the target repre- sentation immediately after the SelfNorm. Then we turn off the chosen SelfNorm and optimize the original image to make its representation fit the target one. In this way, we can examine SelfNorm's 's effect by observing changes in image space. As shown in Figure 66 SelfNorm can primarily reduce the contrast and color the first network block. The effect becomes more subtle as SelfNorm goes deeper into the network. One possible explanation is that the high-level representations lose many low-level details, making it difficult visualize their changes.
In addition to visualizing individual SelfNorms, it is also interesting to see their compound effect. To this end, we reconstruct an image from random noises by matching its representation with a given one. The reconstructed image can show what information is preserved by the feature representation. By comparing two reconstructed images from a network with without SelfNorm, we can observe the joined recalibration effects of SelfNorms before selected location. From Figure [ we can find SelfNorms the first two network blocks can suppress much style information and preserve object shapes. The reconstructions from block 3 do not look visually informative due to the high-level abstraction. Even so, SelfNorms can restrain the high-frequene signals kept in the vanilla network.
8

---

Under review as a conference paper at ICLR 2021
CN in image space
CN at the start of block 1
CN at the end of blockk
Figure 5: CrossNorm visualization at image level (Left), the head (Middle) and tail (Right) of block in a WideResNet-40-2. Both the content (Row) and style (Column) images are from CIFAR-IO. The style rendering changes from global to local as CrossNorm gets deeper in the network.
SN at the start of block
SN the end of block SN at the end of block 2 SN at the end of block
Figure 6: Visualizing single SelfNorms by comparing images before (Top) and after (Bottom) them. The left two, lying in shallow locations, can adjust styles by suppressing color and adding blur. As SelfNorm goes deeper, the recalibration effect subtle, due the high-level feature abstraction.
Plain network
SelfNorm
Plain network
+SelfNorm
Plain network
SelfNorm
Input
The end of block 1
The end of block 2
The end of u'block
Reconstructed images from intermediate CNN features
Figure 7:
Visualizing accumulated SelfNorms by comparing inverted images. SelfNorms in block 1 can wash away much style information preserved in the vanilla network. Similarly, the plain net- work's final representation retains some frequeryy signals which are suppressed by SelfNorms.
In the CrossNorm visualization, pair one content image with multiple style images for better illustration. We first forward them to get their representations at a chosen position. Then, we com- pute the standardized features from the content image representation and the means and variances of the style image representation. The optimization starts from the content image and tries to fit its representation the target one mixing the standardized features with different means and variances. Figure 5 shows diverse style changes made by CrossNorm. The style changes become more local and subtle as CrossNorm moves deeper in the network.
CONCLUSION
this paper, we have presented SelfNorm and CrossNorm, two simple yet effective normalization techniques to improve OOD robustness. They form unity of opposes as they confront and conform to each other terms of approach (statistics usage) and goal (OOD robustness). Beyond their extensive applications, they may also shed light on developing domain agnostic methods applicable to multiple fields such as vision and language, and broad OOD generalization circumstances such unseen corruptions and distribution gaps across datasets. Given the simplicity of SelfNorm and CrossNorm, we believe there is substantial room for improvement. The current channel-wise mean and variance are not optimal encode diverse styles. One possible direction to explore better style representations.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.060 2016.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Ekin D Cubuk, Barret Zoph. Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In CVPR, pp. 113-123, 2019a.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. arXiv preprint 1xXvv:1999.137199 2019b.
Lucas Deecke, Lain Murray. and Hakan Bilen.
Mode normalization.
arXiv preprint
18X.0:1810.05466, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. arXiv preprint arXiv:1610.07629 2016.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. In Proceedings of the International Conference on Learning Representations, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: /903. 12261, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of -ut-of-distribution generalization. arXiv preprini arXiv:2006. 16241, 2020a.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song- Pretrained transformers improve out-of-distribution robustness. arXiv preprint arXiv:2004.06 2020b.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph. Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. In Proceedings of the International Conference on Learning Representations, 2020c.
Daniel Ho, Eric Liang, Ion Stoica, Picter Abbeel, and Xi Chen. Population based augmentation: Efficient learning of augmentation policy schedules. arXiv preprint arXiv: 000.00555999 2019.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-exciiatio networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708, 2017.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance nor- malization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1501- 1510, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.0311 1502.03167, 2015.
10

---

Under review as a conference paper at ICLR 2021
Yoon Kim. Convolutional neural networks for sentence classification. 1408:1408.888... 2014.
arXiv preprint
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurIPS, pp. 1097-1105, 2012.
Xilai Li, Wei Sun, and Tianfu Wu. Attentive normalization. arXiv preprint arXiv: 2019.
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuai Yang. Universal style transfer via feature transforms. Advances neural information processing systems, pp. 386- 396, 2017.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In NeurlPS, pp. 6662-6672, 2019.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 13310,440,55 2015.
Andrew Maas, Raymond Daly, Peter Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th amnual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 5188-5196, 2015.
Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at once: Enhancing learning and generalization capacities via ibn-net. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 464-479, 2018.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European conference on computer vision, pp. 102-118. Springer, 2016.
Evgenia Rusak, Lukas Schott, Roland Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. A simple way to make neural networks robust against diverse image corruptions. arXiv preprint 1xXi::2001.06057. 2020.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Ad- vances in Neural Information Processing Systems, 33, 2020.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
sentiment
Kihyuk Sohn, David Berthelot, Chun-L.ang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.07635 2020.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 1412.6662 2014.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint 16Xiv:1007.0800 2016.

---

Under review as a conference paper at ICLR 2021
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Improved texture networks: Maximizing quality and diversity in forwafo stylization and texture synthesis. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 6924 2017.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEÉ conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Satilovvanni-incenellll Kurt Keutzer, and Bo- qing Gong. Domain randomization and pyramid consistency: Simulation-to-redt generalization without accessing target domain data. In Proceedings of the IEEE International Conference Computer Vision, pp. 2100-21100 2000 2019.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. Interna- tional Conference on Computer Vision (ICCV), 2019.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
arXiv:1605.07146, 2016.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 171.:09//1.0.. 2017.
Ruimao Zhang, Zhanglin Peng, Lingyun Wu, Zhen Li, and Ping Luo. Exemplar normalization for learning deep representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12726-12735, 2020.
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 286-301, 2018.
12

---

Under review as a conference paper at ICLR 2021
SN/CN
SX/CN
Residual
Residual
Residual
Residual
Residual
SN/CN
SN/ON
Original
Identity
Pre-residual
Post-residual
Post-addition
Figure 8: Illustration of SelfNorm (SN) and CrossNorm (CN) positions a residual module.
52
SelfNorn
52
1-instance 2-instance
CrossNorm
50
50
46
46
Ideniity
Pre-residual
Post-residual
Post-addzion
Netthe
Content
Style
Both
(0) mCEs Setifooom and CresNorm residual positions.
(b) mCEs of and 2-instance CrossNom with cropping types.
Figure 9: Study of Modular positions (left) and CrossNorm variants (right). SelfNorm and Cross- Norm both get the lowest errors at the post-addition position. Second, the 2-instance mode consis- tently outperforms the 1-instance one, and proper cropping can decrease the error.
SUPPLEMENTARY
6.1 ABLATION STUDY ON DESIGN CHOICES
All the experiments here use 40-2 Wide ResNet, measured by the mCE on CIFAR-100-C.
CrossNorm variants. CrossNorm can be in l-instance or 2-instance mode with four cropping options. Figure 9 compares their mCEs. The 2-instance mode constantly gets lower errors than the l-instance. Furthermore, cropping can help decrease the eтror since it can encourage the style augmentation diversity. Note that style cropping may not always be superior. We find that best cropping choice may change over backbones, datasets, and joint application with SelfNorm.
Modular positions. Here we investigate four positions in a residual module, illustrated in Figure 8 We can find that, in Figure both SelfNorm and CrossNorm work the best at the post-addition position. For positions other module types, refer to the ablation study in Section 6.2 Further, we find that their order seems to have little influence on the performance, indicated in Table
Blocks choices for SelfNorm and CrossNorm. After narrowing down their positions in a cell, we still need to study which blocks in network should build the cells with them. According to They both perform the best when plugged into all blocks.
Table 6
Ablating components. SelfNorm learns to recalibrate test styles to decrease texture sensitivity, while CrossNorm augments training styles to reduce texture bias. Although focusing on distinct aspects, they lower the corruption error both separately and jointly, according to Table 7 On top of them, cropping and the consistency regularization in AugMix can further advance model robustness. Besides, SelfNorm outperforms SE (Hu et al. 2018 for model robustness because takes advantage of the style information.
6.2 ABLATION STUDY ON CIFAR
In addition to the residual module, we also investigate the positions of SelfNorm and CrossNorm in the cells of AllConvNet, DenseNet, and ResNeXt. Figure 10 illustrates the positions in the All- ConvNet and DenseNet cells. The position results of SelfNorm and CrossNorm on two datasets CIFAR-10-C 10-C and CIFAR-100-C 100-C are given in Tables g
10 and II We also report the results of
SelfNorm and CrossNorm with different cropping choices in Tables 12 and 13 The full component
ablation studies with different backbones are given in Tables 14 and 15
13

---

Under review as a conference paper at ICLR 2021
Table 5: Order of SelfNorm and CrossNorm. this experiment, they both the post-addition position in residual cell.
Order mCE(%)
SN-CN
CN-SN
46.9
46.6
Table 6: Block choices of SelfNorm and CrossNorm. We compare the mCEs (%) when applying SeifNorm and CrossNorm to image space different blocks in WideResNeet40-2. The choice of all blocks gives the lowest errors for both of them.
Stages
N/A
Image
Block 1
Block 2
Block 3
All
SelfNorm
53.3
52.9
48.9
52.2
51.3
47.4
CrossNorm
53.3
54.3
52.2
51.2
51.5
48.8
Table 7: Ablation study of SE, SelfNorm(SN), (rossNorm(CN)) cropping, and consistency regular- (CR), SN obtains much lower corruption error than SE, justifying its robustness superiority. Besides, SN and CN can work together in reducing mCE, and cropping and CR can help further.
Basic
SE
SE(post)
SN
CN
SNCN SNCN+Crop SNCN+Crop+CR
mCE(%) 53.3
52.3
51.0
47.4
48.8
46.6
44.5
43.7
6.3 VISUALIZATION CONTINUED
Visualization setup. Our visualization builds on the technique: understanding deep image repre- sentations by inverting them (Mahendran & Vedaldi 2015). The goal is to find an image whose feature representation best matches the given one. The search done automatically by SGD opti- mizer with learning rate 1e4, momentum 0.9, and 200 iterations. The learning rate is divided by 10 every 40 iterations. During the optimization, the network is in its evaluation mode with its parame- ters fixed. In the experiment, we use WideResNNet-4-22 and images from CIFAR-10. In visualizing CrossNorm, we use the training images and a model trained for epoch. The SelfNorm visualization uses test images and a well-trained model. We use different settings for them because CrossNorm is for training, while SelfNorm works in testing.
More visualization results. Figure extending Figure 5 shows more CrossNorm visualizations in deeper network blocks. Figure 12 gives an illustration of corruptions used in robustness evalu- ation on CIFAR and ImageNet. Moreover, Figure 13 some synthetic images from GTA5 and realistic ones from Cityscapes. The visualization of CrossNorm applied to the synthetic images is provided in Figure 14
6.4 ABLATION STUDY ON IMAGENET
Table 16 reports the results of applying SelfNorm CrossNorm with IBN. We see that they can cooperate to improve corruption robustness. Moreover, in Tables 17 and 18 we also investigate
the SelfNorm and CrossNorm positions in a residual module using ImageNet. Similar the CIFAR results, the post-addition position performs the best for corruption robustness.
We also compare CrossNorm to Imageeet,,, which transfers styles from external datasets to perform style augmentation. Stylized- ImageNee finetunes pre-trained ResNet-50 for 45 epochs with double data (stylized and original ImageNets) in each epoch. To compare CrossNorm with Eeylized-ImageNer
perform the finetuning for 90 epochs using only the original ImageNet. In Table 19 although Stylized-ImageNet has 2% lower corruption егтоr than CrossNorm. its clean is 3.8% higher. Because the external styles in Stylized-ImageNe cause large distribution shifts, impairing its clean generalization. In contrast, The more consistent yet diverse internal styles help CrossNorm decreases both corruption and clean errors.
14

---

Under review as a conference paper at ICLR 2021
Convolution
Convolution
Convolution
Convolution
Convolution
SN/CN
Norm Layer
Norm Layer
SN/CN
Norm Layer
SN CN
GELU
SN CN
GELUU
GELU
SN / CN
Pre
Post
2
SN and CN positions in AllConvNet
SN and CN positions in dense cell
Figure 10: Illustration of SelfNorm (SN) and CrossNorm (CN) positions in AllConvNet block, and dense cells in DenseNet. For blocks AllConvNet, we name the position after convolution layer as 1, after normalization layer as 2, and after GELU layer as 3. For dense cells in DenseNet, we label the position before feature concatenation as Pre, and after concatenation Post.
6.5 ABLATION STUDY ON GENERALIZATION FROM SYNTHETIC TO REALISTIC DATA
In addition to Table 3 Table 20 provides the ablation study on cropping for domain generalization from GTA5 Cityscapes.
Table 8: Ablation study of the impact of different SelfNorm position for AllConvNet, DenseNet, WIdeResNet and ResNeXt on CIFAR-10-C measured by mCE
Position
AllConvNet 24.01
26.38
25.56
Position
Pre
Post
DenseNet
23.40
21.96
Position
Residual
Post
Pre
Identity
WideResNet
22.69
21.29
20.78
22.29
Position
Residual
Post 24.76
Pre 21.49
Identity
ResNeXt
21.94
21.99
Table 9: Ablation study of the impact of different CrossNorm position for AllConyNet, DenseNet, WideResNet and ResNeXt CIFAR-10-C measured by mCE
Position
AllConyNet 25.99
26.27
26.84
Position
Pre
Post
DenseNet
24.72
29.17
Position
Residual
Post
Pre 24.91
Identity
WideResNet
25.20
21.62
23.27
Position
Residual 26.71
Post
Pre 23.75
Identity 26.92
ResNeXt
22.37
15

---

Under review as a conference paper at ICLR 2021
Table 10: Ablation study of the impact of different SelfNorm position for AllConyNet, DenseNet, WIdeResNet and ResNeXt LEFAR-100-C measured by mCE
Position
AllConyNet
50.30
51.63
51.03
Position
Pre
Post
DenseNet
53,86
54.67
Position
Residual
Post
Identity
WideResNet
49.28
47.44
49.82
48.43
Position
Residual 47.56
Post
Pre 50.86
Identity 50.44
ResNeXt
49.00
Table 11: Ablation study of the impact different CrossNorm position for AllConvNet, DenseNet, WIdeResNet and ResNeXt on CIFAR-100-C measured by mCE
Position
AllConvNet
52.20
52.49
52.69
Position
Pre
Post
DenseNet
55.44
57.57
Position
Residual
Post
Pre
Identity
WideResNet
52.06
48.76
51.74
50.26
Position
Residual 51.47
Post
Pre
Identity 50.22
ResNeXt
46.95
47.87
Table 12: Ablation study of the impact of SelfNorm + CrossNorm with different cropping, including neither, content, style and both, for AllConvNet, DenseNet, WIdeResNet and ResNeXt on CIFAR- 10-C measured by mCE
Backbone
Neither
Content
Style
Both
AllConvNet,
18.98
20.29
18.82
20.28
DenseNet, Conv Pre
18.75
18.23
18.70
18.83
WideResNet, Post
17.93
17.99
16.77
17.47
ResNeXt, Post
17.73
18.52
18.37
18.59
Table 13: Ablation study of the impact of SelíNorm + CrossNorm with different cropping, including neither, content, style and both, for AllConvNet, DenseNet, WIdeResNet and ResNeXt on CIFAR- 100-C measured by mCE
Backbone
Neither
Content
Style
Both
AllConvNet,
44.17
46.92
43.86
46.08
DenseNet, Convl Pre
51.38
49.40
49.13
48.97
WideResNet. Post
46.58
45.08
45.82
44.46
ResNeXt, Post
41.03
44.85
42.96
46.52
16

---

Under review as a conference paper at ICLR 2021
Table 14: Ablation study of each component of our methods, including SelfNorm, CrossNorm with
cropping, consistency and combination with AugMix, for AllConvNet, DenseNet, WIdeResNet and ResNeXt on CIFAR-IO-C measured by mCE
Backbone
Basic
SN
CN
SN+CN
SN+CN+Crop +Consistency
AugMix
SNCN+Crop
+Crop
+AugMix
Alt,onvNet, 1, style
30.80
24.01
25.99
18.82
17.24
15.00
11.79
DenseNet, Convl Pre, both
30.70
21.96
24.72
18.83
18.53
12.7
10.40
WideResNet, Post, both
26.90
20.78
21.62
17.47
16.93
11.2
9.94
XtsNeXt, Post, neither
27.50
21.49
22.37
17.73
15.69
10.90
9.09
Table 15: Ablation study of each component of our methods, including SelfNorm, CrossNorm with cropping, consistency and combination with AugMix, for AllConvNet, DenseNet, WIdeResNet and ResNeXt on CIFAR-I00-C measured by mCE
Backbone
Basic
CN
SN+CN
SN+CN+Crop +Consistency
AugMix
SNCN+Crop
+Crop
+AugMix
AllConvNet, 1, style
56.40
50.30
52.20
43.86
42.83
42.7
36.80
DenseNet, Convl Pre, both
59.30
53.86
55.44
48.97
48.48
39.60
36.95
WideResNet, Post, both
53.30
47.44
48.76
44.46
43.70
35.90
33.38
ResNeXt, Post, neither
53.40
47.56
46.95
41.03
40.84
34.9
30.76
Table 16: Ablation study of IBN, SelfNorm(SN), CrossNorm(CNN, consistency regularization(CR) and AugMix(AM) on ImageNet-C with ResNet50, IBN, originally designed for domain generaliza- tion, can also decrease mCE. Both SN and CN can further lower the error based IBN. Combining them with AM gives the best robustness performance.
ResNet50
ResNet50+1BN(a))
ResNet50+1BN(()
Basic
Basie
+CN+CC+++CCCCCC +CN+AM
Basic
+SN
+SN+AM +NCN++M
Clean err(%)
23.9
23.2
23.1
22.6
22.5
24.0
23.5
22.3
22.3
mCE(%%
80.6
75.1
73.2
73.6
66.4
74.1
72.6
64.1
62.8
Table 17: Position investigation of SelfNorm in residual module of ResNet50 trained 90 epochs on ImageNet.
Modular Position
Identity
Pre-Resee Pre-Residual
Post-Residuse
Post-Addition
Clean error (%)
24.0
23.0
23.2
23.7
mCE(%)
75.5
75.8
74.8
73.4
Table 18: Position investigation of CrossNorm residual module of ResNet50 trained 90 epochs on ImageNet.
Modular Position
Identity
Rre-Residual
Dost-Residual
Post-Addition
Clean error (%)
25.2
23.4
23.5
23.4
mCE(%)
78.2
75.8
77.5
75.3
(SIN), SeI)Norm(SN), CrossNorm(CN), and crop- ping with ResNet50 trained 90 epochs on ImageNet. Compared with SIN, CN holds a better balance between clean and corruption emror.. SN also works well in decreasing the corruption enror while maintaining low clean error. Combining SN and CN can further lower both errors.
Table 19; Ablation study of
Basic
SIN
SN
CN
SNCN
SNCN+Crop
Clean error (%)
23.9
27.2
23.7
23.4
23.3
23.3
mCE(%)
80.6
73.3
73.4
75.3
70.4
71.1
17

---

Under review as a conference paper at ICLR 2021
Table 20: Ablation on GTAV-Cityscapes domain generalization. Mean IoU for both within-domain and cross-domain evaluation reported. All methods use FCN with ResNet50 as backbone net- work. We use style-only crop for segmentation.
Methods
FCN Baseline
SN
CN
CN+Crop
SNCN
SNCN+Crop
Source
63.7
64.6
61.8
61.2
65.0
63.5
Target
21.4
29.9
31.5
32.0
34.1
36.5
Style images
Style images
Style images
CN the start block 2
CN at the end of block 2
CN at the start of block 3
Figure 11: CrossNorm visualization at the head (Left), the tail of (Middle) block 2 and the start of block 3 (Right) in WidcResNet-40-2. Both the content (Row) and style (Column) images are from -0 Compared to CrossNorms blocks and have weaker style transfer effects. Because the channel-wise means and variances in high-level feature maps may contain less low-level visual information.
block 1, shown in Figure [ the CrossNorms in
Gaussian Noise
Shot Noise
Impulse Noise
Defocus Blur Frosted Glass Blu
Motion Blur
Zoom Blur
Snow
Frost
Fog
Brightness
Contrast
Elastic
Pixelate
JPEG
Figure 12: A demostration of corrupted images in ImageNet- C dataset (Hendrycks & Dietterich 2019). 15 types of algorithmically generated corruptions from noise, blur, weather, and digital categories are applied to images to create corrupted dataset.
18

---

Under review as a conference paper at ICLR 2021
Some examples of segmentation dataset. The first row are images from Cityscapes dataset, while the second row are images from GTAS dataset.
Figure 13:
Figure 14: A visualization of CromsNorm with crop style used on image level. The two images on the first column the original images in the GTA5 dataset and are in the same training batch. We applied CrossNorm to these two images several times and got the following three pairs of images. By calculating style from random crops, CrossNorm can perform variety of style augmentation.1 It should be noted that we use CrossNorm in both image and feature level, we only visualize in image level here for simplicity.
19


Reviews:
[{"review_id": "uqa9HExYji7", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary\n\nThe paper presents two new methods to improve corruption robustness and domain generalization: SelfNorm, a way to adapt style information during inference, and CrossNorm, a simple data augmentation technique diversifying image style in feature space. Both methods are tested on Cifar10/100-C, ImageNet-C, Semi-Supervised Cifar and GTA V -> Cityscapes.\n\n\n2. Strengths\n+ The approach is straight forward and apparently very effective\n+ The evaluation is performed reasonably\n+ The method is tested not only on corruptions but also on domain adaptation. This is great as both tasks can be seen as two sides of the same metal and it is great to see more methods testing on both.\n\n3. Weaknesses\n- The main problem is the write up. In the introduction \"texture sensitivity\" is introduced as a concept but never picked up later on. In general the motivation is very high level drawing a lot on the concepts of style, texture and content but the method itself is rather down to earth modifying instance normalization parameters. In my opinion the high level arguments and colorful terms (style, shape sensitivity, unity of opposites etc.) distract rather than add to the story. They may serve as inspirations but without extensive experiments it is hard to related concepts like style to the manipulation of network parameters. \n- The ablation study was hard to follow and tbh I think it could have been shortened presenting only the most important results in the paper and the rest in the appendix.\n- Conversely I felt the paper would profit from more figures and visualizations (e.g. Figure 7). The method is very simple and I think that is a major strength. It should thus also be possible to present it in a more concise form.\n\n\n4. Recommendation \n\nAs it is right now I think the paper has to be rejected because the write up is just too chaotic and vague. I do however like the method a lot and I would vote for accept if it was rewritten substantially to focus more on an understandable presentation of the method than abstract concepts and colorful terms. \n\n\n4. Questions/Recommendations\n- It is probably beyond the scope of the review period but a demonstration of the technique on some non image data would be amazing making the theoretical argument in Section 4.1 that the method can work on other domains much more powerful.\n- It would be nice to include a discussion of \"Improving robustness against common corruptions by covariate shift adaptation\".\n\n6. Additional feedback \n- Figures 1 and 2 are a bit hard to understand.\n- IMHO it would be more interesting to include Figure 7 in the paper and move Figure 3 into the appendix.\n- The use of IBN should be mentioned in Table 2", "extra_scores": {}, "extra_text": {}}, {"review_id": "XZdvHHyWizO", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\n\nThis paper proposes two novel norms: Selfnorm and Crossnorm for model robustness. Selfnorm recalibrates style in features to reduce texture sensitivity. Crossnorm performs style augmentation (swap channel-wise means and variances) to reduce texture bias. The authors also give some discussions on the distinction and connection between SelfNorm and CrossNorm. The proposed norms can be applied to different settings and tasks. Their method shows state-of-the-art robustness performance on both fully and semi-supervised settings, and classification and segmentation tasks.\n\n**Clarity**\n\n- The paper is well organized and easy to read.\n\n**Strengths**\n\n- Contributions clearly stated and validated.\n- Comprehensive experiments to show the effectiveness of their method. \n- The proposed method is domain agnostic and can be applied to different settings and tasks.\n\n**Weaknesses**\n\n- SelfNorm is a learned normalization, thus I think the statement \"they take opposite actions ... at separate stages testing v.s. training ...\" (page 1) is confused. Should SelfNorm be used in both testing and training?\n\n- In the experiments, both SelfNorm and CrossNorm units are placed in a ResNet block. What is the order of their placement? It's not clear in the paper.\n\n- It is better to keep the original formula (as shown in Figure 1(a)) in Eq.2 as well so that it is easy for readers to understand.\n\n- The related residual channel attention (CA) [a] should be discussed in this paper. Because the residual channel attention can be also considered as a normalization that recalibrates/rescales features with channel-wise attention. The authors are suggested to give some discussion on the difference between the CA mechanism and proposed Norms. CA is also a unit that is inserted in the ResNet block so that it's interesting to have an experimental comparison as well.\n\n[a] Zhang at al., Image Super-Resolution Using Very Deep Residual Channel Attention Networks, in ECCV'18\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DaNjwapGtn", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThe paper introduces two simple modules, SelfNorm (SN) and CrossNorm (CN), that are highly modular and can theoretically be attached to different parts of the CNNs to control the balance between style and content cues for their recognition. SN is a lightly parametrised module that gives freedom to the CNN to dynamically re-adjust the means and variances of the feature maps in the original instance normalization framework. The authors argue that SN then learns to emphasize _important styles_ and suppresses less important ones, with the underlying assumption that the first and second-order statistics are often sufficient and necessary representations for style. CN is a twin invention that randomly swaps the mean and variance statistics of features of two channels. This leads to diversified virtual styles in the training data, effectively factoring out the model's dependence on style cues for recognition, according to the authors.\n\n**Pros**\nIt is great news to the field that such simple recipes introduce gains in the performances. From the industrial point of view, the expansion of choices for model design means the likelihood of introducing uniform gains across the board in industrial applications (think about how much $ worth it will be if it cuts the error rate by 0.1 pp across all image classifier applications). From the research point of view, it is exciting to confirm yet again in 2020 that there is much room for improvement in those modular components, after the introduction of SE blocks and the likes.\n\n**Cons**\nThe above excitement should be checked with a pinch of salt. I find it difficult to follow the rationale behind key assumptions (e.g. that shape = content and texture = style) and the experimental section is, I have to admit, a bit chaotic. And I suspect if the chaos is intended, for downplaying the discouraging side of the results.\n\n1. **Shape=content and texture=style? No.**\nFirst of all, we need a decent, if not mathematical, definition of the four terms above. To me, \"content\" is the causal cue that constitutes the GT label for the task at hand; I refer to all the rest cues as \"style\". \"Texture\" to me is a local pattern that can be captured by sliding windows of, say, size 10x10 pixels and \"shape\" is any pattern that is more global than texture. Under this terminology, I find it hard to agree that texture is not content. Texture does contribute, as the causal cue, to the final task at hand in many real-world computer vision applications - texture classification, fine-grained cat categorization, medical diagnosis with CT scans, and semantic parsing for detecting snowy and watery road conditions, to name a few. \n\n2. **The oversimplistic view of the world that underlies this paper makes the justifications for the proposed methods all the more fragile.**\nIt is difficult to agree that SN and CN, which are argued to control style and content for the benefit of the recognition task at hand, are really working as speculated. It is also disputable whether the first and second-order statistics really encode the style and/or texture component. I find it a bit dangerous to let such a convenient and simplifying view of the matters be published and guide researchers in the field to adopt the same kind of viewpoint. In the revision, try to introduce more depths in your arguments and include more empirical analysis to make a point SN and CN work in the promised way.\n\n3. **Introduce order in the experimental reports.**\nSo I made a table below to summarize the experimental setups in this paper (sorry for the dots everywhere - formatting tricks ;)). I'm having a hard time convincing myself that SN and CN really work empirically, given those highly specific choices of settings per task and analysis. For example, why is ImageNet performance for SN and CN not compared against augmentation baselines considered in the CIFAR experiments in Tab1? What are the individual performances of CN and SN for CIFAR on those 4 architectures in Tab1? What is the effect of location for SN in a CNN (equivalent analysis for CN is presented in Tab4)? There are many, many questions unanswered. I do observe a few improvements introduced by the two modules here and there, but I can't forgo the impression that these are only selected highlights that comply with the authors' arguments. Please introduce the much-needed order in the experimental section, and only then will I be able to assess if the new technology is truly innovative.\n| .  Section  .  | .  Data  . | .  Arch  . | .  Evaluation  . | .  Baselines  . | .  Authors' methods  . | \n| -- | -- | -- | -- | -- | -- |\n| .  Tab1  . | .  CIFAR  . | .  4 archs  . | .  mCE,CleanAcc  . | .  Cutout,Mixup,Cutmix,AA,Advtr,AugMix  . | .  SNCN, SNCN+AugMix |\n| .  Fig2  . | .  CIFAR  . | .  28-2WideResNet  . | .  mCE,CleanAcc  . | .  WA,RA  . | .  CN |\n| .  Fig4  . | .  CIFAR  . | .  40-2WideResNet  . | .  mCE  . | .  VanillaModel  . | .  SN,CN |\n| .  Tab4  . | .  CIFAR  . | .  40-2WideResNet  . | .  mCE  . | .  VanillaModel  . | . CN |\n| .  Tab5  . | .  CIFAR  . | .  40-2WideResNet  . | .  mCE  . | .  VanillaModel  . | . SN,CN,SNCN,SNCN+Crop,SNCN+Crop+CR |\n| .  Tab2  . | .  ImageNet  . | .  ResNet50 . | .  mCE,CleanAcc  . | .  PU,AA,MaxBlur,SIN,AugMix  . | . CN,SN,SNCN+AugMix |\n\n**Key reasons for the rating**\nThe simplicity and effectiveness of the technologies argued by the authors are eclipsed by the oversimplifying assumptions and inefficient experimental exposition. The rating reflects this disappointment. Please aim to improve the paper in the rebuttals and paper revisions.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8_MpViQfCVF", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the CNN model robustness against problems, e.g., texture sensitivity and bias. In particular, this paper proposes to recalibrate style using SelfNorm motivated by the fact that attention help emphasize essential styles and suppress trivial ones and reduce texture bias using CrossNorm by swapping feature maps within one instance. The opposite two processes forms a unity of using style to advance model robustness. The intuition of the proposed method is clearly presented and the paper is well structured. Detailed comments are summarized as follows:\n\nPros:\n\t- The paper contributes a solution by forming a unity of opposites in using style for model robustness. Both the motivation and intuition are clearly presented. The idea is somewhat novel for me.\n\t- The method can be easily implemented and integrated into classical frameworks.\n\t- Extensive experiments under different settings and tasks show the effectiveness of the proposed method.\n\nCons:\n\t- In Sec. 3 Unity of Opposites, the authors explains that SelfNorm works during testing while CrossNorm functions only in training. The statement cannot clearly illustrates the process in training and testing. I suppose both SelfNorm and CrossNorm would be used during training to get a trade-off point and only SelfNorm is used during testing. It is encourage to add an algorithm flowchart to indicate the process.\n\t- The authors explains SelfNorm recalibrate feature style while  CrossNorm performs style augmentation. It is suggested to visualize the attention/feature maps before and after using the proposed techniques to illustrate their benefits.\nThe experiment results of SNCN are not significant compared with AugMix though their combination achieves new state-of-the-art.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
 This paper proposes two mechanisms, SelfNorm (used during training and inference) leveraging an attention-based recalibration of mean and standard deviation for instance normalization, and CrossNorm which performs cross-channel swapping of mean/stdev. Is is shown that the combination (often combined with AugMix) performs well across several datasets in terms of model robustness. Overall the paper has strength in the fact that the method is interesting, simple to implement, and modular. However, reviewers brought up a number of issues including the overstated motivation/writing, lack of clarity, and most importantly need for clear experimental results (comparing to uniform/standard baselines) and identification of the separate mechanisms. It is especially uncertain why it is necessary that they are used *together* (often with AuxMix as well) to obtain the strong performance. As a result, the score for this paper is borderline, tending towards a weak acceptance. 

It is appreciated that the authors provided a lengthy rebuttal, including new results in a different domain (NLP); however, the reviewers agreed that not all of the concerns were addressed.  After a lengthy discussion, all of the reviewers agree that while the method is simple, modular, and effective when combined (hence the positive scores from some reviewers), the authors fail to describe the underlying reason for the method's gains, especially with respect to the individual parts (SelfNorm vs. CrossNorm) and why the results only come when these rather independently derived modules are used together. The exposition of the experimental results, with differing baselines/conditions that make it very hard to understand where the effect is coming from, exacerbates this issue. 

As a result of these concerns, I recommend rejection of this paper. However, the method is interesting and results promising, so I hope that the authors can clarify the writing and improve the presentation of the results (specifically separating out the effects of SelfNorm and CrossNorm, as well as analyzing how they interact together to improve results) and submit to a future venue. 
Decision: Reject