DYNAMIC BATCH NORM STATISTICS UPDATE FOR NATURAL ROBUSTNESS
Authors: Shahbaz Rezaei
Mohammad Sadegh Norouzzadeh
Venue: Submitted to ICLR 2023 (2023)

Abstract: DNNs trained on natural clean samples have been shown to perform poorly on corrupted samples,
such as noisy or blurry images. Various data augmentation methods have been recently proposed
to improve DNN’s robustness against common corruptions. Despite their success, they require
computationally expensive training and cannot be applied to off-the-shelf trained models. Recently,
updating only BatchNorm(BN) statistics of a model on a single corruption has been shown to improve
its accuracy on that corruption significantly. However, adopting the idea at inference time when the
type of corruption changes decreases the effectiveness of this method. In this paper, we harness the
Fourier domain to detect the corruption type, a challenging task in the image domain. We propose
a unified framework consisting of a corruption-detection model and BN statistics update that can
improve the corruption accuracy of any off-the-shelf trained model. We benchmark our framework
on different models and datasets. Our results demonstrate about 8% and 4% accuracy improvement
on CIFAR10-C and ImageNet-C, respectively. Furthermore, our framework can further improve the
accuracy of state-of-the-art robust models, such as AugMix and DeepAug.

Under review as a conference paper at ICLR 2022
DYNAMIC BATCH NORM STATISTICS UPDATE FOR NATURAL ROBUSTNESS
Anonymous authors Paper under double-bling review
ABSTRACT
DNNs trained on natural clean samples have been shown to perform poorly on corrupted samples, such as noisy or blurry images. Various data augmentation methods have been recently proposed to improve DNN's robustness against common corruptions. Despite their success, they require computationally expensive training and cannot be applied to off-the-shelf trained models. Recently, updating only BatchNorm (BN) statistics of model on a single corruption has been shown to improve its accuracy on that corupption significantly. However, adopting the idea at inference time when the type of corruption changes decreases the effectiveness of this method. In this paper, we harness the Fourier domain to detect the corruption type, a challenging task in the image domain. We propose unified framework consisting of corruption-detectio model and BN statistics update that can improve the corruption accuracy of any off-the-shelf trained model. We benchmark our framework on different models and datasets. Our results demonstrate about 8% and 4% accuracy improvement on CIFARI 10-C and ImageNet-C, respectively. Furthermore, our framework can further improve the accuracy of state-of-the-ie robust models, such as AugMix and DeepAug.
INTRODUCTION
Deep neural networks (DNNs) have been successfully applied to solve various vision tasks in recent years. At inference time, DNNs generally perform well data points sampled from the same distribution as the training data. However, they often perform poorly on data points of different distribution, including corrupted data, such noisy blurred images. These corruptions often appear naturally at inference time in many real-world applications, such as cameras autonomous cars, x-ray images, etc. Not only DNNs accuracy drops across shifts in the data distribution, but also the well-known overconfidence problem of DNNs impedes the detection of domain shift.
One straightforward approach to improve the robustness against various corruptions is to augment the training data to cover various corruptions. Recently, many more advanced data augmentation schemes have also been proposed and shown to improve the model robustness on corrupted data, such as SIN Geirhos et al. (2018a), ANT Rusak et al. (2020a), AugMix Hendrycks et al. (2019), and DecpAug Hendrycks et al. (2021). Despite their efffcctiveness, these approaches require computationally expensive training or re-training process.
Two recent works (Benz et al., 2021; Schneider al., 2020) proposed a simple batch normalization (BN) statistics update to improve the robustness of pre-trained model against various corruptions with minimal computational overhead. The idea is to only update the BN statisties of pre-trained model on a target corruption. If the corruption type is unknown beforehand, the model can keep BNs updating inference time to adapt the ongoing corruption. Despite its effectiveness, this approach only suitable when constant flow of inputs with the same type of corruption is fed to the model so that it can adjust the BN stats accordingly.
In this work, we first investigate how complex the corruption type detection task itself would be. Although corruption type detection is challenging in the image domain, employing the Fourier domain make it much more manageable because each corruption has relatively unique frequency profile. We show that a very simple DNN can modestly detect corruption types when fed with a specifically normalized frequency spectrum.

---

Under review as a conference paper at ICLR 2022
Given the ability to detect corruption types in the Fourier domain, adopt the BN statistic update method such that can change the BN values dynamically based on the detected corruption type. The overall architecture of approach is depicted in Figure 1. First, calculate the Fourier transform of the input image, and after applying specifically designed normalization, it is fed to the corruption type detection DNN. Based the detected corruption, fetch the corresponding BN statistics from the BN stat lookup table, and the vre-trained network BNs are updated accordingly. Finally, the dynamically updated pre-trained network processes the original input image.
In summary, our contributions are as follows:
We harness the frequency spectrum of image to identify the corruption type. On ImageNet- C, shallow 3-layer fully connected neural network can identify 16 different corruption types with 65 58.88 accuracy. The majority of the misclassifications occur between similar corruptions, such as different types of noise, for which the BN stat updates are similar nevertheless.
Our framework be used any off-the-shelf pre-trained model, even robustly trained models, such as AugMix Hendrycks et al. (2019) and DeepAug Hendrycks et al. (2021), and further improves the robustness.
We demonstrate that updating BN statisties inference time suggested in (Benz et al., 2021; Schneider et al., 2020) does not achieve good performance when the corruption type does not continue to be the same for long time. On the other hand, framework is insensitive to the rate of corruption changes and outperforms these methods when dealing with dynamic corruption changes.
METHOD
2.1
OVERALL FRAMEWORK
The overview of our framework is depicted in Figure 1. It consists of three main modules: A) a pre-trained model on the original task, such as object detection, B) DNN trained to detect corruption type, and C) lookup table storing BN statistics corresponding to each type of corruption. This paper mainly focuses improving the natural robustness of trained DNNs. However, the framework can be easily extended to domain generalization and circumstances where the lookup table may update the entire model weights or even the model architecture itself.
Corruption Type Detection
BN stats Lookup
Fourler Spectrum jonily amplitude)
Update the model
Pre trained Network
Class labels
2
Figure 1: Overall Framework
2.2 ADAPTATION TO NEW CORRUPTIONS
In (Benz et al., 2021; Schneider et al., 2020), a simple BN statistic update has significantly improved the natural robustness of trained DNNs. Figure show the effectiveness of their approach on various corruption types. The drawback of their approach is that the BN statistics obtained for one type of corruption often significantly degrades the accuracy for other types of corruption, except for similar corruption, such as different types of noise. The authors claim that in many applications, such as autonomous vehicles, the corruption type will remain the same for a considerable amount of time. Consequently, the BN statisties can be updated inference time. However, neither of those papers
2

---

Under review as a conference paper at ICLR 2022
10
44
14
0.12
13
4.81
0.7
455
2.
4.0
070
41
45
-33
73
-3
motoon Dkr
2.0
0.26
3.
42
344
13
8.4
81
52
37
13
gausser rotre
34
115
13
13
-37
6.2
74
4.6
6.7
33
37
47
-3
12
0.0
notoe
35
-15
-13
-0
3.6
4.3
-27
0.44
4.25
28
9.89
2.7
4.5
or Mftinaaa
-23
-14
-13
-.2
-15
2,670
2.13
0,022
00.13
4.64
-A
1.
0.94
4.7
4.9
-.2
4.3
3.
4A
4.7
2.9
-3A
per_coorpestion
22
1.3
0009
ORboU Mir
000 Mar
STOW
Figure 2: ResNet18 (ImageNet-C)) The y-axis shows the corruption with which the model BN stats updated. The x-axis shows the corruption which the model performance is evaluated. The numbers in the cells are accuracy gain compared to the original model, the model with BN stats obtained from the natural dataset.
has shown the performance of BN statistic update when the corruption type changes. We conduct an experiment in Section 3.4 show that detecting corruption types and utilizing appropriate BN stats provides better results when the corruption type is not fixed.
2.3 CORRUPTION DETECTION
The average Fourier spectrum of different corruptions has been shown to have different visual appearances Yin et al. (2019). However, conducting a corruption classification on the Fourier spectrum of individual images is not trivial task. Feeding DNN with the raw Fourier spectrum leads to poor results and unstable training. Here, we first visually investigate the Fourier spectrum of various corruption types. Then, we propose tailored normalization technique and shallow DNN to detect corruption types.
We denote image of size (21,d2) by x E Rdixdd,, We omit the channel dimension here because the Fourier spectrum of all channels turns out to be similar, when the average is taken over all samples. We only show the results of the first channel here. We denote natural and corrupted data distribution by D, and De, respectively. We denote 2D discrete Fourier transform operation by F. In this paper, we only consider the amplitude component of since the phase component does help much with corruption detection. Moreover, we shift the low-frequency component to the center for better visualization.
Figure shows the normalized Fourier spectrum of different corruption types in CIFARIO-C. The results on ImageNet-C is presented in Figure 4. We explain the normalization process in the next paragraph. For visualization purposes, we clamp the values above one. However, we do not clamp pixel values of the input when fed to the corruption detection model. As shown in Figure 3, most corruption types have distinguishable average Fourier spectrum. The almost identical ones, i.e., different types of noise, are not needed to be distinguished accurately because the BN stat updates for one of them improve the accuracy for others nevertheless, shown in Figure 2.
3

---

Under review as a conference paper at ICLR 2022
(6) Defocus blur
(b) Motion blur
(c) Zoom blur
(d) Glass blur
(c) JPEG
Shot noise
(e) Gaussian
(h) Impuise
Pixelate
( Blastic
noise
moise
form
(K) Brightness
()) Contrast
(m) Fog
(n) Frost
(0) Snow
Figure Normalized Fourier spectrum of CIFARIO-C dataset
(a) Defocus blur
(b) Motion blur
(c) Zoom blur
(d) Glass blur
(c) JPEG
Shot noise
Gaussian
(h) Impulse
Pixelate
Elastic
noise
noise
form
(k) Brightness
(l) Contrast
(m) Fog
(n) Frost
(o) Snow
Figure 4: Normalized Fourier spectrum of ImageNet-C dataset
To normalize the data, we first obtain the average Fourier spectrum of the natural samples, denoted by en EE-DD, [=)[] Then compute normalized Fourier spectrum by log( [==._11F(a)) + 1) for each corruption type, separately. For corruption detection purpose, substitute the expected value over the entire corruption type dataset by an individual image, i.c.. log( [F(=)] empirically find this specific normalization to outperform others significantly. The intuition behind this normalization is twofold: First, natural images have a higher concentration in low frequencies Yin et al. (2019). Although corrupted images also have large values on low-frequency components, they may also have large concentration
6n
c.. + 1). We
high-frequene components, depending on the corruption. Hence, we divide the values by 6n to ensure that model does not exclusively focus on low-frequency components during training. Second, the range of values from one pixel to another may vary multiple orders of magnitude, which causes instability during training. Typical normalization techniques on unbounded data, such as tanh sigmoid transforms, leads to poor accuracy because values larger than certain point converge to and become indistinguishable.
We employ three-layer fully connected (FC) neural network for corruption-type detection. Despite having an image-like structure, we avoid using convolutional neural networks (CNNs) here because of the apparent absence of shift- thift-invariance in the Fourier spectrum. Due the symmetry in the Fourier spectrum, we only feed half of the Fourier spectrum to the model. For CIFAR1O, we flatten the 2D data and feed three-layers FC model with 1024, 512, and 16 neurons. Note that this
4

---

Under review as a conference paper at ICLR 2022
AEI KMA FM
Molle san
M4
60
600
600
600
COL CO
CX1
640
645
660
666
666
666
645
64A 666 666
600
C00
600 6O
CM
140
640
640
330
334
vwA. MI M0
M41
60) KM 66 MO 60 K01
3M
KAI EOC B4G 640 G41
GOL
E60 600
640
(a) CIFARIO-C -49.21% accuracy
(b) ImageNet-C
-65.8% accuracy
Figure 5: Corruption type detection model's confusion matrix
paper deals with 15 corruption types and natural data previously experimented in (Benz et al., 2021; Schneider et al., 2020). For ImageNet-C. we first use 2D average pooling with kernel size and stride of 2 to reduce the input size. Then, flatten the output and feed it model with three FC layers of size 2058, 512, and 16. Additionally, we use ReLU function as non-linearity after the first and second layers.
We train the model with stochastic gradient descent (SGD) for 50 epochs. We decrease the learning rate by factor of 10 epochs 20 and 35. We only 100 samples per corrppiion/nteenity, and we keep the rest for validation. Using the Fourier spectrum and the proposed normalization method, we achieve validation accuracy of 49.21 % and 65.88% on CIFAR10-C and ImageNet-C, respectively. The same architecture and model capacity only yields 7.64% and 6.32% accuracy in the image domain. We also could not achieve good accuracy with CNNs in the image domain. The confusion matrix of the corruption detection is presented in Figure
small number of samples for training, i.e.,
3 EXPERIMENTAL. SETUP
Datasets & Metries. CIFARIO dataset Krizhevsky al. (2009) contains 32 x 32 color images of 10 classes, with 50,000 training samples and 10,000 test samples. ImageNet dataset Deng et al. (2009) contains around 1.2 millions images of 1000 classes. For ImageNet, we resize images to 256 x 256 and take the center 224 x 224 as input. CIFARIO-C and ImageNet-C datasets Hendrycks & Dietterich (2019) contain corrupted test samples of the original CIFARIO 10 and ImageNet. There are 15 test corruptions and 4 hold-out corruptions. For a fair comparison with previous work, we only use the 15 test corruptions as in (Benz et al., 2021; Schneider ct al., 2020). Each corruption type, c, contains 5 different intensities severity level, denoted by Similar to Hendrycks et al. (2019), we use unnormalized corruption error uCE = Σ Ee,, on CIFARIO, and normalized corruption error CE = -S E E. EAlerNet for ImageNet-C. Corruption error averaged over all 15 corruptions is denoted by mCE.
Models. Our framework consists of two DNNs, namely the corruption type detector and a pre-trained model on the original task. The details of the corruption type detector model are explained in Section 2.3. For CIFAR1O, we consider 200, ResNet- 10 He et al. (2016), VGG-19 Simonyan & Zisserman (2014), WideRes Zagoruyko & Komodakis (2016), and DenseNet (L=100,, k=12)
5

---

Under review as a conference paper at ICLR 2022
Table 1: Evaluation results on CIFARIO-C
Model
All combined (aceuncy)
On natural images (aceanccy)
On compped images (aceamey)
On cormupia images (mKCE)
Bose
Ours
A
Base
Ours
A
Base
Ours
A
Basc
Ours
A
20
69.4616
16.82%
1.37%
91.62%
90.85%
-0.76%
67.98%
15,899
7.91%
32,02%
24.11%
-7.91%
ResNes-11
72.06%
80.59%
8.53%
93.55%
93.01%
-0.54%
70.63%
79.768
9.13%
29.37%
20.24%
-9.13%
VGG-19
74.08%
6.61%
92.95%
-0.24%
72.8198
7.07%
27.19%
-7.0%%
WRN 28-12-811 10
78.00%
85.21%
1.22%
96.23%
96.05%
-0.18%
76.78%
84.89%
7.71%
23.22%
15.51%
-7.1%%
DenseNer
74.34%
81.915
7.58%
95.0456
94,49%
-0,5556
72,9096
81,0798
8.12%
27.04%
18.90%
-8.12%
Table 2: Per corruption accuracy on CIFARIO-C
Model
Blur
Noise
Weather
Digital
Brightass
Defbeus
Glass
Shot
Impalse
Ftost
Snow
Eig
Contratt
Elusc
ResNet
75,95
08.46
68.96
45.07
55.00
43.11
35.73
70.65
75.28
82,73
89.87
67.96
77.06
07.88
74,.45
RasNet-20
83.00
02.33
79.28
77.35
79.79
ResNet: 110
79.33
73.00
73.67
43.30
55,10
56.33
73.18
78.26
$7.10
91,82
95.71
81.18
77.02
(ours)
62.78
84.47
82.72
VGG-19
46.59
77.
VGG--19 (ours)
86.77
84.55
64.57
67.29
63.93
84
87.95
91.75
77.67
84.72
83.05
WRN-28-10
59.87
54.18
90.84
77.91
WRN-25 10 (ouns)
90.43
87.02
39.44
71.77
73.43
72.89
71.63
88.78
87.67
92:37
94.79
88.53
81.72
85.17
81.08
DenseNet
82.16
78.54
75.52
54.83
56.73
44.44
46.42
78.79
$2.73
88.77
93.20
79.27
84.23
71.29
76.90
DenseNer (omrs)
87.93
85.09
64.82
68.32
72.56
65.80
65.87
85.78
85.53
90.53
93.16
86.98
84.85
82,00
76.85
Huang et al. (2017). All CIFARIÓ models are adopted from a public github repository' For ImageNet, we consider ResNet-18, ResNet-50 He (2016), VGG-19 Simonyan & Zisserman (2014), WideResNet-500 Zagoruyko & Komodakis (2016), DenseNet-16 161 Huang et al. (2017). All ImageNet models are adopted from torchvision library Marcel & Rodriguez (2010). We also adopted trained ResNet-50 models from state-of-the-an robustness literature, i... Stylized ImageNet training (SIN) Geirhos et al. (2018a), adversarial noise training (ANT) Rusak al. (2020a), AugMix Hendrycks et al. (2019), and DeepAug Hendrycks et al. (2021).
BN Statistics. In this paper, we specifically adopted BN stat update from Schneider et al. (2020) with 1. For corruption this choice of parameters indicates that take an
parameters N 1 and average of a natural BN stats and the BN stats of the corruption c. We compute BN stats from the same samples we use to train the corruption-type detection model. Due the small sample size for BN stat adoption, we find that taking an average with natural BN stats leads to better results than only using the target corruption BN stats.
3.1 EVALUATION ON CIFARIO-C
Table presents the results of CIFAR 10-C over several models. Our approach improves the accuracy all corruptions by around 8%. However, the accuracy over natural samples is dropped by less than 1%. Because the base model is trained on natural samples, any misclassification of natural samples in the corruption detection model negatively affects the model performance, while any correct classification of corruptions positively affects the accuracy. As shown in Table 2, our approach significantly improves the accuracy over all the corruption types, except for brightness and JPEG corruption, in which the accuracy barely changes. Note that these two corruptions have the least improvement when BN stat applied, as shown Figure 2.
3.2 EVALUATION ON IMAGENET-C
Evaluation results on ImageNet-C is shown in Table 3 and 4. We observe similar pattern as CIFARIO with slightly smaller improvement. Here, accuracy improvement is around 4%. Similarly, improvement occurs over all corruptions except for brightness and JPEG.

---

Under review as a conference paper at ICLR 2022
Table 3: Evaluation results on ImageNet-C
Model
All combiiee (securacy)
On natural images (securacy)
Cln cormpptedd insges (sccuracy)
Ou corragted images (mCE)
Base
Ours
Bise
Ours
A
Buse
Onrs
A
Base
Ours
ResNet-11 18
34.00%
37.94%
3.88%
69.2616
68.57%
-1.18%
31.68%
35.90%
4.22%
86.63%
81.48%
-5.15%
ResNet:50
40.4%%
44.42%
3.94%
76.13%
75.04%
-1.09%
38.10%
42.38%
4.28%
78.43%
73.13%
-5.29%
74.2216
73.42%
78.84%
WRN-50
44.33%
47.20%
2.87%
78.47%
77.03%
-1.40%
42.06%
45.21%
3.15%
73.31%
69.50%
-3.81%
DenseNer
47.57%
50,08%
2,52%
77,1498
76.56%
40.58%
45.59%
48.32%
2.72%
68.88%
65.5318
-3.35%
Table 4: Per corruption accuracy on ImageNet-C
Model
Blur
Nosse
Weather
Digital
Driphnnes
Defocus
Moton
Zoo8
Shot
Causs
Ingpu
Stow
Flank
Pxolace
KesNee-8 18
28,01
29.02
20.N1
22.67
24.10
58,85
42.13
40.25
(ours)
50
36.35
31.12
26.59
30.5
65.19
35.95
45.55
52.44
ResNes (ours)
36.47
40.39
38.39
29.77
35.69
38.04
33.71
38.83
37.44
48.70
03.
42,87
44.87
51.77
$3.60
VGG-19
28.63
25.19
23.71
32,10
57.75
48.07
VGG- 19 (ours)
29.94
34.91
32:43
24.10
30.54
32.73
31. 31.98
36.43
35.18
46 85
62.76
39.01
39.36
45,13
47.96
WRY-50
19.35
40.08
38.48
28.55
35.88
37.98
33.12
38.65
32.35
44.55
67.47
38.45
46. 16
$2.92
56.87
WRN-50 (ours)
36,60
42,38
38.45
50379
43.45
45.56
40.98
42.75
401.26
48.33
66.
44.58
47.06
55.45
55.06
DenseNet
40,09
40.12
38.42
29.58
40.95
42.31
37.18
43.84
39.64
52.13
09. 00.80
50.01
47.42
$4.53
57.89
DenseNet (ours)
38.98
42.64
40.26
45.31
47.37
43.11
56.34
69.35
52.96
47.37
58.33
58.85
3.3 EVALUATION ON ROBUST MODELS
In this section, investigate if our approach can further improve the accuracy of state-of-the- -frttr-rrtttttt models on IeageNet-C. Table 5 presents the evaluation of five state-of-the-arr models. Our approach consistently improves the performance of robust approaches even further. Note that here exclude the data we use to train the corruption type detection model from the validation set. That explains the small discrepancy between the base accuracy reported in the paper and those in previous work.
3.4 INFERENCE TIME ADAPTATION
Two recent papers (Benz et al., 2021; Schneider et al., 2020) that investigated BN statistics update suggested that the idea can be used at inference time, and the model will adopt new corruption eventually. However, they have never empirically evaluated their performance for inference time adaptation. Here, start with the original model trained on clean samples. Then, during evaluation, after a certain number of batches, we randomly pick another corruption and then continue evaluating the model. The samples within samples each batch. We let the model BN stats be updated from the last ten batches at the beginning of each batch. Because our approach does not update the BN stat lookup table, is insensitive how the inference time evaluation is conducted, and consequently, the performance is similar.
batch come from only single corruption, and there are 16
The results of the experiment shown in Figure 6. In CIFARIO, only in VGG-19 and only when we let the corruption stay the same for 32 consecutive batches our approach is underperformed. In ImageNet, both VGG-19 and ResNet18 outperforms our approach only after 32 successive
Table 5: Accuracy of ResNet50 on ImageNet-C
Model resnet50
Basc
Our adaptation 42.38%
A
38. 10%
4.28%
resnet50 SIN Geirhos et al. (2018a)
37.78%
40.29%
2.51%
resnet50 ANT+SIN Rusak et al. (2020a)
46.63%
48.71%
2.08%
resnet50 AugMix Hendrycks al. (2019)
46.96%
50.44%
3.47%
resnet50 DeepAug Hendrycks et al. (2021)
51.47%
53.18%
1.71%
resnet50 AuugAug+AugMix Hendrycks al. (2021)
55.67%
59.12%
3.45%
7

---

Under review as a conference paper at ICLR 2022
AACU (50
Medkit LtD
Rodke Scieedlo
Vileo3
WEBL-SS Screicer
20
Numberd of consecutive batches wits samte comuption
Number of consecueive baiches aith carre corrupt
(a) CIFARIO-C
(b) ImageNet-C
Figure 6: BN stat update vs. our framework at inference time. The x-axis shows the number of consecutive batches for which the corruption remains the same during the evaluation. It takes many consecutive batches of the same corruption for models catch up the corruption change when inference time BN stat update is deployed. Our framework, however, is insensitive to the corruption changes and can adapt instantly.
batches. This experiment reveals that the original BN stat update mechanism in (Benz al., 2021; Schneider et al., 2020) only works when input corruption remains the same for a considerable number of consecutive samples. Although this assumption is reasonable for some applications, such as autonomous vehicles with continuous stream input, does not hold for many applications, particularly for non-stream inputs common in healthcare applications.
4 LIMITATIONS & DISCUSSION
One major limitation of the current framework is that it needs data samples from all corruption types to train the corruption type detection model. Although using the Fourier spectrum allows us to train the corruption detector easily with small number of samples, it still limits the generalizability of the framework to unseen corruptions. One solution to this problem is to attach an off- the--hheeeff outlier detection mechanism
uncertainty mechanism to discover time. Then, we can make a new entry in the BN stat lookup table, and the model can gradually learn BN statistics inference time by observing multiple samples from the new class. Hence, can prevent the need to collect image samples from all corruptions during training. Another related perspective is frame the supervised corruption type detection an unsupervised problem. This reformulation is possible because the corruption labels themselves are nonessential in our framework. For example, we can use a clustering algorithm to cluster different corruption and then associate each cluster with an entry in the BN stats table. This strategy can also be extended to detect new clusters inference time for better generalization. We will investigate this idea in future work.
types of corruption inference
In this paper, Our framework is only evaluated natural and corrupted images. We can employ the same corruption detection idea for domain detection. Since the pre-trained model does not need to be re-trained in our framework, it might be interesting adopt framework for domain generalization. For instance, a natural image and cartoon have distinguishable features, such as color distributions, Fourier spectrum, etc. Accurate domain detection might be simple task if proper features are found.
Currently, our framework accuracy is bounded by the BN statistics update proposed in (Benz et al., 2021: Schneider al., 2020). As result, with the presence of perfect corruptionddmmain detection, the accuracy may not be improved the BN statistic update does not work for the target corruption/domain. In the future, will investigate other approaches eliminate this limitation.
8

---

Under review as a conference paper at ICLR 2022
5 RELATED WORK
Dodge et al. Dodge & Karam (2017) revealed that deep models' accuracy significantly drops with corrupted images despite having similar performance to humans on clean data. Several studies (Geirhos et al., 2018b; Vasiljevic et al., 2016) verified that training with some corruptions does not improve the accuracy for unseen corruptions. However, Rusak et al. (2020b) later challenged this notion by showing that Gaussian data augmentation can enhance the accuracy of some other corrupiions as well. In (Benz et al., 2021; Schneider et al., 2020), authors have shown that corruption accuracy can be significantly increased by only updating the BN statistics of trained model on specific corruption. Although it claimed that can be easily adopted at inference time by updating the model BN stats using batch of most recent samples, the performance of the models has not been evaluated in a situation where the corruption type changes.
There are numerous data augmentation methods shown to improve corruption robustness. AutoAug- ment Cubuk et al. (2019) automatically searches for improved data augmentation policies but has been shown later to improve coruppiio eπror Yin et al. (2019). AugMix Hendrycks et al. (2019) combines a set of transforms with a regularization term based on the Jensen-Shannon divergence. has been shown that applying Gaussian noise image patches also improve accuracy I Lopes et al. (2019). In Hlylized-ImageNee the idea of using style-transfer were adopted for data augmentation Geirhos al. (2018a). Using adversarially learned noise distribution has been proposed in Rusak et al. (2020a). In DeepAug Hendrycks et al. (2021), images are passed through image-to-image models while being distorted to create new images leading to large improvements in robustness. The adoption of adversarially training to improve corruption robustness has not been consistent. For instance, Rusak et al. (2020b) has shown that adversarial training does not improve corruption robustness while Shen et al. (2021) and Ford et al. (2019) have reported otherwise, using loo adversarial training.
6 CONCLUSION
In this paper, we propose framework where an off-the-shelf naturally trained vision model can be adapted to perform better against corrupted inputs. Our framework consists of three main components: 1) corruption type detector, 2) BN stats lookup table, and 3) off-the-shelf trained model. Upon detecting the corruption type with the first component, our framework pulls the corresponding BN stats from the lookup table and substitutes the BN stats of the trained model. Then, the original image fed to the updated trained model.
Even though detecting the corruption type is a very challenging task in the image domain, the Fourier spectrum of an image to detect the type of corruption. We use a shallow three-layer FC neural network that detects the corruption type based on Fourier amplitudes of the input. We show that this model can achieve significant accuracy by training on minimal samples. The same small sample size is shown be also enough to obtain the BN stats stored in the BN stat lookup table.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Philipp Benz, Chaoning Zhang, Adil Karjauv, and In So Kweon. Revisiting batch normalization for improving corruption robustness. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 494-503, 2021.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 113-123, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Samuel Dodge and Lina Karam. A study and comparison of human and deep learning recogni- tion performance under visual distortions. In 2017 26th international conference computer communication and networks (ICCCN), pp. 1-7. IEEE, 2017.
Nic Ford, Justin Gilmer, Nicolas Carlini, and Dogus Cubuk. Adversarial examples are a natural consequence of test error in noise. arXiv preprint 1XXXv:9991..0//// 2019.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trainee trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: 1.188111.222313 12231, 2018a.
Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. Advances in neural information processing systems, 31, 2018b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: /2261, 2019.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv: met::092220000 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of -uttof-distriuution generalization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8340-8349, 2021.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4705 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin D Cubuk. Improving robustness without sacrificing accuracy with patch gaussian augmentation. arXiv preprint arXiv:1906.02611 02611, 2019.
Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In Proceedings of the 18th ACM international conference on Multimedia, pp. 1485-1488, 2010.
Evgenia Rusak, Lukas Schott, Roland Zimmermann, Julian Bitterwolfb, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. Increasing the robustness of dnns against im-age corruptions by playing the game of noise. 2020a.
Evgenia Rusak, Lukas Schott, Roland S Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. A simple way to make neural networks robust against diverse image corruptions. In European Conference Computer Vision, pp. 53 $3.69. Springer, 2020b.
10

---

Under review as a conference paper at ICLR 2022
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann. Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in Neural Information Processing Systems, 33:1
2020.
Yu Shen, Laura Zheng, Manli Shu, Weizi Li, Tom Goldstein, and Ming Lin. Gradient-free adversarial training against image corruption for learning-base steering, Advances in Neural Information Processing Systems, 34:26250-26266 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 2014.
Igor Vasiljevic, Ayan Chakrabarti, and Gregory Shakhnarovich. Examining the impact of blur recognition by convolutional networks. arXiv preprint 1.05760 2016.
Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective model robustness in computer vision. Advances in Neural Information Processing Systems, 32, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv: 1605.07166. 2016.


Reviews:
[{"review_id": "aGfqdQUFiZ", "reviewer": "Reviewer_Jakz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper seeks to improve the robustness of DNNs against common visual corruption, e.g., snow, fog, and blur. The authors propose to identify the type of corruption by leveraging the Fourier spectrum, and update the BN stats with a pre-computed look-up table. The experimental results are provided on top of ResNet.", "summary_of_the_review": "I think the current paper is not ready to be published on ICLR. See the weaknesses above.", "strength_and_weaknesses": "Strength:\n- The studied problem is of practical importance.\n\nWeaknesses:\n- Many state-of-the-art deep networks, such as vision Transformers, do not leverage batch normalization. The proposed method can not be applied to them.\n- Most realistic corruption may not be able to be categorized into one of a small number of pre-defined classes.\n- The novelty of this paper may be limited. The proposed method is more like an engineering solution, which integrates some existing techniques. The idea of identifying corruption in the frequency domain has been widely explored in the field of low-level vision.\n- The writing of this paper requires improvement.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "_aNb2at-8kF", "reviewer": "Reviewer_s7TA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript proposes a way to update BatchNorm statistics to improve the robustness of dynamic corruptions at inference time. Specifically, the proposed method detects corruptions at the Fourier domain and adaptively updates BN statistics based on the detected corruption types. The extensive experiments showed the effectiveness of the proposed method on the robustness benchmarks such as CIFAR-10C and ImageNet-C.", "summary_of_the_review": "Overall, I recommend rejection for the following reasons;\n- At the high level, the proposed method is motivated to handle more realistic scenarios on the model robustness, however, it further requires more strict and unrealistic assumptions such as corruption supervision, and all corruption types are known in the training phase.\n- The provided explanations and experimental results are not enough to clearly understand the proposed method and its effectiveness (e.g., weak baselines, lack of explanations, as I raised above)\n- This work has limited novelty, as I raised above. \n\n", "strength_and_weaknesses": "Strengths\n- The writing is clear and easy to understand\n- This manuscript introduces more realistic scenarios, which handle dynamic corruption during the testing phase, than the prior works\n- The experimental results are strong compared to the baseline.\n\nWeaknesses\n- Strong assumptions and Gaps between the tackled problem and the proposed method; To overcome the drawback of the prior works, which assume a single corruption type at the inference phase, the authors target the case dynamic corruption types given at the inference phase. However, as the authors discussed in the Limitation section, the proposed method requires several strong assumptions on the problem; the authors assume that (1) they have all the corruption supervision at the training phase to train the detector model, (2) there are only seen corruptions at the inference phase, and (3) the testing samples are sorted in order to have the same corruption types in consecutive batches, not randomly at the inference phase. Furthermore, guessing from the lack of explanation, it seems to assume to know when the corruption types change and know what it is at the inference phase without using the detector. This doubt comes from Figure 6, which shows the fixed performance over different testing setups despite the performance of the detector is not high (nearly 50% on CIFAR-10C).\n- Weak technical novelty; all components of the proposed method are from the literature [1,2,3]. The benefits of using the Fourier domain or solely updating BN statistics for robustness are empirically known in the recent literature. The authors also just followed BN updating rules from the literature. \n- Weak baselines; overall, this manuscript does not provide enough baselines to evaluate the significance of the proposed method. There are no baselines handling robustness (e.g., [1,2]) in Tables 2 and 3 (which show the effectiveness of the single corruptions given at the inference). Similarly, it is difficult to understand the effectiveness of the proposed Fourier detector; Are the accuracies of 49% on CIFAR-10C and 65% on ImageNet-C high? There are no comparisons between Fourier-based methods like [1].\n- Lack of explanation: There are no clues on how to obtain the BN lookup Table. How many samples are used to obtain the corruption-specific BNs? How do we use the trained detector to obtain the lookup Table?\n\n[1] Benz et al., \"Revisiting batch normalization for improving corruption robustness,\" ICCV 2021\n[2] Schneider et al., \"Improving robustness against common corruptions by covariate shift adaptation,\" NeurIPS 2020\n[3] Yin et al., \"A Fourier Perspective on Model Robustness in Computer Vision,\" NeurIPS 2019\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TjZVWcGXWk", "reviewer": "Reviewer_uu4e", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "An approach is proposed for improving performance when corruption types are known in advance, and when one cannot or is not willing to retrain the underlying model. The approach involves 1. training a small corruption-type classifier that operates on carefully-normalized, frequency-domain input images, 2. a table of batch norm statistics for each corruption type, and 3. an existing network that heavily leverages batch normalization (this network does not need to be retrained, but rather has its BN stats updated based on the corruption type, as determined by the corruption-type classifier). This method is evaluated only on the same types of corruptions which undergo the same 15 corruption types that were used to train the corruption-type classifier, using CIFAR10-c and ImageNet-C, standard corrupted variants of CIFAR and ImageNet. In this setting, the proposed approach is carried out using a variety of common networks (ResNet-18, ResNet-50, DenseNet, etc.), and shown to generally perform better on corruption inputs (+ approx. 8% for CIFAR, + approx. 4% for ImageNet) and a bit worse on non-corrupted images (- approx. 0.5% for CIFAR, - approx 1% for ImageNet).", "summary_of_the_review": "The experiments show that the proposed solution 'works', but I can't recommend acceptance primarily because of the weaknesses above, and in particular because of the problem itself. My biggest suggestion to the authors is to try to strongly motivate the problem of mitigating effects of known input corruption types while being unable to retrain the underlying model.", "strength_and_weaknesses": "The biggest strength of the paper is that it may be practically useful in some settings (in particular those that may be subject to the specific types of corruptions considered (jpeg compression, motion blur, snow, etc.). The authors show that, if these same corruptions are encountered, then this approach can generally improve performance without the need to retrain the original model (which is more difficult to retrain than the small corruption-type classifier). In addition, the authors are transparent about the limitations in Section 4.\n\nThe biggest weaknesses of the paper are that 1. insights for the reader are limited and 2. the problem being solved -- mitigating effects only of *known* types of input corruption, while further restricted to be unable to retrain the underlying network -- is too restrictive, and not well motivated. With regard to (1), many interesting bits were already tackled in prior work, e.g. proposing batch norm updates for robustness to corruption (e.g. Benz et al. 2021), showing that frequency magnitude is indicative of corruption type (e.g. Figures 3 and 4 are very similar to this in Yin et al. 2019). The method described for frequency-domain normalization, before the corruption-type classifier, does have merit, but at the same time it is a simple heuristic, and in addition it's never tested through an ablation study. With regard to (2), I admire the authors for highlighting the obvious limitation of 'known corruption' in Section 4, but it is nevertheless a significant limitation. Further, it is not at all clear that the proposed directions for alleviating this limitation would work (outlier detection or unsupervised learning for learning new corruption types), and, even if one could possibly get this to work – wouldn't it be easier in most cases to just retrain, or more realistically fine tune, the underlying models?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8_F5r4fOAf", "reviewer": "Reviewer_1hRd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper builds on previous batch norm adaptation results, in which statistics are computed from a larger batch of test images. Instead, the authors build a \"domain predictor\" based on frequency features, and then learn conditional statistics for each of the domains. The method is benchmarked on CIFAR-C and ImageNet-C.", "summary_of_the_review": "While the approach is interesting and might be worth pursuing, the paper lacks rigour in the evaluation and the discussion of the results within the existing literature. Specifically the main claim about the usefulness of the Fourier space to distinguish corruption domains is not sufficiently supported by experiments.", "strength_and_weaknesses": "I opted for a relatively short summary of the main weaknesses, as it does not make sense to discuss the paper in more detail before fixing/commenting these, and I am looking forward to discussing with the authors.\n\n**Major Weaknesses**\n\n- The literature is not sufficiently discussed. E.g. the first entry paragraph does not back up any sentence with citations, and the following paragraphs are also quite sparse in citations.\n- The paper is lacking a discussion of more powerful methods than Batch norm adaptation: What about more recent test-time adaptation methods like TENT (Wang et al., 2020) or EATA (Niu et al., 2022)?\n- \"[...] detect the corruption type, a challenging task in the image domain\": While I like the authors' approach to use the Fourier spectrum, this sentence seems to be wrong. Any one-layer convnet with sufficiently large kernel can learn to perform a Fourier transform, so arguably also if the \"domain detector\" is trained in \"image space\", the network could discover this stategy. The authors should provide experimental evidence that this sentence is true, or remove it from the abstract.\n- The proposed method based on the Fourier spectrum is surprisingly bad (cf. Figure 5). Especially the noise types are confused. This shows an inherent limitation of the method, as e.g. distinguishing shot and gaussian noise is trivial in the image domain (shot noise has extreme values at 0/255 and can be easily detected in the histogram).\n\n**Minor Weaknesses:**\n\n- The citations are not correctly typeset (using `\\cite` instead of `\\citep` at most places)\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject