Blur Is an Ensemble: Spatial Smoothings to Improve Accuracy, Uncertainty, and Robustness
Authors: Namuk Park
Songkuk Kim
Venue: ICLR 2022 Submitted (2022)
Keywords: bayesian neural network
uncertainty
uncertainty estimation
uncertainty quantification

Abstract: Bayesian neural networks (BNNs) have shown success in the areas of uncertainty estimation and robustness. However, a crucial challenge prohibits their use in practice. Bayesian NNs require a large number of predictions to produce reliable results, leading to a significant increase in computational cost. To alleviate this issue, we propose spatial smoothing, a method that ensembles neighboring feature map points of CNNs. By simply adding a few blur layers to the models, we empirically show that spatial smoothing improves accuracy, uncertainty estimation, and robustness of BNNs across a whole range of ensemble sizes. In particular, BNNs incorporating spatial smoothing achieve high predictive performance merely with a handful of ensembles. Moreover, this method also can be applied to canonical deterministic neural networks to improve the performances. A number of evidences suggest that the improvements can be attributed to the stabilized feature maps and the flattening of the loss landscape. In addition, we provide a fundamental explanation for prior works—namely, global average pooling, pre-activation, and ReLU6—by addressing them as special cases of spatial smoothing. These not only enhance accuracy, but also improve uncertainty estimation and robustness by making the loss landscape smoother in the same manner as spatial smoothing. 


Under review as a conference paper at ICLR 2022
BLUR IS AN ENSEMBLE:
SPATIAL SMOOTHINGS TO IMPROVE ACCURACY, UNCERTAINTY, AND ROBUSTNESS
Anonymous authors Paper under double-blind review
ABSTRACT
Bayesian neural networks (BNNs) have shown success the areas of uncertainty estimation and robustness. However, a crucial challenge prohibits their use in practice. Bayesian NNs require a large number of predictions to produce reliable results, leading to significant increase in computational cost. To alleviate this issue, we propose spatial smoothing, method that ensembles neighboring feature map points of CNNs. By simply adding few blur layers the models, empirically show that spatial smoothing improves accuracy, uncertainty estimation, and robustness of BNNs across whole range of ensemble sizes. In particular, BNNs incorporating spatial smoothing achieve high predictive performance merely with handful of ensembles. Moreover, this method also can be applied to canonical deterministic neural networks to improve the performances. A number of evidences suggest that the improvements can be attributed the stabilized feature maps and the flattening of the loss landscape. In addition, we provide fundamental explanation for prior works works-namely, global average pooling, artivativa,ion, and ReLU6-by addressing them as special cases of spatial smoothing. These not only enhance accuracy, also improve uncertainty estimation and robustness by making the loss landscape smoother in the same manner spatial smoothing.
5
7
11
12
13
14
16
18
INTRODUCTION
19 In a real-world environment where many unexpected
20 events occur, machine learning systems cannot be guar-
2.0
Defeminiatic
Deterministic Smooth drapout MC dropout Smooth
21 anteed always produce accurate predictions. In or- der to handle this issue, we make system decisions
VOR
VGG-19
1.8%
23 more reliable by considering estimated uncertainties, in addition predictions. Uncertainty quantification is
1.2
NLL
25 particularly crucial in building a trustworthy system in the field of safety-critical applications, including med- ical analysis and autonomous vehicle control. However, canonical deep neural networks (NNs)-or determinis- produce reliable estimations of uncer- tainties (Guo et al., 2017), and their accuracy often severely compromised by natural data corruptions from
VGG-19
VGG-16
1.0
27
ResNet-18 ResNet-50
28
ResNet-F -15
29
tic
0.8
ResNet-S0
30
70.0
77.5
31
Accuracy (%)
32 noise, blur, and weather changes (Engstrom et al., 2019; Azulay & Weiss, 2019).
Figure 1: Spatial smoothing improves both accuracy and uncertainty (NLL). Smooth means spatial smoothing. Down- ward from left to the right (*)) means bet- ter accuracy and uncertainty.
34 Bayesian neural networks (BNNs), such as Monte Carlo (MC) dropout (Gal & Ghahramani, 2016), provide a probabilistic representation of NN weights. They com- bine number of models selected based weight prob- ability to make predictions of desired results. Thanks to this feature. BNNs have been widely used in the areas of uncertainty estimation (Kendall & Gal, 2017) and robustness (Ovadia et al., 2019). They are also promising in other fields like distriibtttiiitoonnn detection (Malinin & Gales, 2018) and meta-learning (Yoon al., 2018).
35
36
38
40

---

Under review as a conference paper at ICLR 2022
Observed data
Temporally proximate data stream
Spatially proximate data points
t=0
t= 2
Input
Po
Po
P1
P
Pn
Prediction
Importance
I/N
INN
BNN
Temporal smoothing
Spatial smoothing
Figure 2: Comparison of three different Bayesian neural network inferences: inference, temporal smoothing (Park et al., 2021), and spatial smoothing (ours). In this figure, 2g is observed data, Ps is predictions p(y æu. w.) or p(y x., w:), n. is importances π (ax|22)) and N is ensemble size.
canonical BNN
42 Nevertheless, there remains a significant challenge that prohibits their use in practice. BNNs require ensemble size of up fifty to achieve high predictive performance, which results in a fiftyfold 44 increase in computational cost (Kendall & Gal, 2017; Loquercio et al., 2020). Therefore, BNNs achieve high predictive performance merely with handful of ensembles, they could be applied 46 to much wider range of areas.
47
1.1
PRELIMINARY
We would first like to discuss BNN inference in detail, then move on to fector-Quantize BNN (VQ-BNN) inference (Park et al., 2021), an efficient approximated BNN inference.
49
BNN inference. Suppose we have access to posterior probability of NN weight pl w D) for training dataset D. The predictive result of BNN is given by the following predictive distribution:
/ p(y æo, w) pDw)D) dw
plyla, D)
where æn is observed input data vector, y output vector, and p jr w) is the probabilistic prediction parameterized by the result of NN for input æ and weight wr In most cases, the integral cannot be solved analytically. Thus, we use the MC estimator to approximate it as follows:
plyl D)
p(y æo. w.)
N
50 where w; ~ p(w ID and N is the number the samples. The equation indicates that BNN inference ensemble average of NN predictions for observed data point as shown the left of Fig. 2. 52 Using neural networks the ensemble would requires N times more computational complexity than one NN execution.
2

---

84 these methods by addressing them as special cases of spatial smoothing. Experiments support the Under review as a conference paper at ICLR 2022
1.00
78.0
NLL
Accuracy
ECE (%)
0.90
0
0--0-
76.0
4.0
0.80
10"
10
10"
10'
10"
10'
Ensemble Size
Ensemble Size
Ensemble Size
Deterministic
Deterministic Smooth
MC dropout
MC drepout Smooth
Figure 3: Spatial smoothing improves both accuracy and uncertainty across a whole range of ensemble sizes. We report the predictive performance of ResNet-18 on CIFAR-I00.
Data-complemented BNN inference. To reduce the computational cost BNN inference, VQ- BNN (Park et al., 2021) executes NN for an observed data only once and complements the result with previously calculated predictions for other data. If we have access to previous predictions, the computational performance of VQ-BNN becomes comparable to that of one NN execution. To be specific, VQ-BNN inference is:
N-1 Σ
xp D)
r ((, xo) p(y/ w.)
(3)
where It (2;; (ma) is the importance of data m; with respect to the observed data æn, and is defined as similarity between æ; and Zt- p(y Do. We) is the newly calculated prediction, and {p(ylæ: æ1, wi).... 56 are previously calculated predictions. To accurately infer the results, the previous predictions should consist of predictions for "data similar to the observed data".
55
57
Thanks to the temporal consistency of real-world data streams, aggregating predictions for similar data in data streams is straightforward. Since temporally proximate data sequences tend to be similar, we can memorize recent predictions and calculates their average using exponentially decreasing importance. In other words, VQ-BNN inference for data streams is simply temporal smoothing of recent predictions as shown in the middle of Fig.
59
60
81
62
63
VQ-BNN has two limitations, although it may be promising approach to obtain reliable results 64 an efficient way. First, was only applicable to data streams such as video sequences. Applying VQ-BNN to images is challenging because it is impossible to memorize all similar images in advance. Second, Park et (2021) used VQ-BNN only the testing phase, not the training phase. We find that ensembling predictions for similar data helps in NN training by smoothing the loss landscape.
66
1.2 MAIN CONTRIBUTION
69
Spatially neighboring points in visual imagery tend to be similar, as do feature maps of convolu- tional neural networks (CNNs). By exploiting this spatial consistency, we propose spatial smoothing as a method of ensembling nearby feature maps to improve the efficiency of ensemble size in BNN inference. The right side of Fig. 2 visualizes spatial smoothing aggregating neighboring feature maps.
70
71
72
73
We empirically demonstrate that spatial smoothing improves the efficiency in vision tasks, such image classification on CIFAR (Krizhevsky et al., 2009) and ImageNet (Russakovsky et al., 2015), without any additional training parameters. Figure shows that negative log- likelloold (NLL) of "MC dropout + spatial smoothing" with an ensemble size of two is comparable to that of vanilla MC dropout with an ensemble size of fifty. We also demonstrate that spatial smoothing improves accuracy, uncertainty, and robustness all the same time. Figure/ shows that spatial smoothing improves both the accuracy and uncertainty of various deterministic and Bayesian NNs with an ensemble size of
74
75
76
77
78
80 fifty on CIFAR-100 100.
Global average pooling (GAP) (Lin et al., 2014; Zhou et al., 2016), pre-activation (He et al., 82 2016b), and ReLU6 (Krizhevsky & Hinton, 2010; Sandler et al., 2018) have been widely used in vision tasks. However, their motives largely justified by the experiments. We provide explanation for
85 claim by showing that the methods improve not only accuracy but also uncertainty and robustness.
3

---

Under review as a conference paper at ICLR 2022
2
PROBABILISTIC SPATIAL SMOOTHING
87
To improve the computational perfor- mance of BNN inference, VQ-BNN (Park et al., 2021) executes NN pre- diction only once and complements
88
89
Stage i
90
91
the result with previously calculated predictions. The key to the success 92
Stage
HxMxxC
Prob (tanh-ReLU)
H*WWCC
Smooth
of this approach largely depends on
Blur (AvgPool)
94 the collection of previous predictions for proximate data. Gathering tempo- rally proximate data and their predic- tions from data streams is easy be- cause recent data and predictions can be aggregated using temporal consis-
Stage 1-1
WH/2XM/2x2C
Stage 1-1
96
H//2K/2K2C
98
Figure 4: Stages of CNNs such as ResNet (left) and the stages incorporating spatial smoothing layer (right).
99
100
tency. On the other hand, gathering independdnn proximate data, c.g.
101
images, is more difficult because they lack such consistency.
102
2.1
MODULE ARCHITECTURE FOR ENSEMBLING NEIGHBORING FEATURE MAP POINTS
103
So instead of temporal consistency, we use spatial consistency- where neighboring pixels of images are similar-for real-world images. Under this hypothesis, we take the feature maps as predictions and aggregate neighboring feature maps.
104
105
Most CNN architectures, including ResNet, consist of multiple stages that begin with increasing the number of channels while reducing the spatial dimension of the input volume. We decompose an entire BNN inference into several steps by rewriting each stage in recurrence relation as follows:
p(zi+1lz;, D) = p(z)+1lz;, w:) pD dw,
(4)
106
where z; is input volume of the i-th stage, and the first and the last volume are input data and output. wa and p(w;(D) are NN weight in the i-th stage and its probability, p( //z, w;) is output probability of æ;+1 with respect the input volume z,. To derive the probability from the output feature map, we transform each point of the feature map into a Bernoulli distribution. To do so, composition of tanh and ReLU, a function from value of range [-o0, so] into probability, is added after each stage. Put shortly, we use neural networks for point-wiee wise binary feature classification.
107
108
109
110
112
Since Eq. (4) is a kind of BNN inference, it can be approximated using Eq. (3). other words, each stage predicts feature map points only once and complements predictions with similar feature maps. Under spatial consistency, it averages probabilities of spatially neighboring feature map points, which is well known blur operation in image processing. For the sake of implementation simplicity, average pooling with a kernel size of 2 and a stride of is used as a box blur. This operation ensembles four neighboring probabilities with the same importances.
114
116
In summary, as shown in Fig. 4, we propose the following probabilistic spatial smoothing layer:
Smooth (z) = Blur Prob (z)
118 where Prob(.) (.) is point-wise function from a feature map to probability, and Blur() is importance- weighted average for ensembling spatially neighboring probabilities from feature maps. Smooth layer added after each stage. Prob and Blur are further elaborated below.
120
Prob: Feature map to probability. Prob a function that transforms a real-valued feature map into probability. We use tanh-ReLU composition for this purpose. However, tanh is commonly known to suffer from the vanishing gradient problem. To alleviate this issue, we propose the following lemperature-scaled tanh:
tanh, (z) = 7 tanh (z/T)
(6)
121 where 7 is a hyperparameter called temperature. J is in conventional tanh and oo in identity function. tanh, imposes an upper bound on value, but does not limit the upper bound to 1.
4

---

Under review as a conference paper at ICLR 2022
Feature map variance
Feature map variance
0.20
0.100
0.15
0.075
0.10
0.050
0.025
0.05
0.000
0.00
c1
a1
c2
s2
c3 Blocks
83 c4
24
G1
81
c2
82
c3
83
84
Blocks
(a) Model uncertainty
(b) Data uncertainty
MC dropout
MC dropout Smooth
Figure 5: Spatial smoothing layers reduce feature map variances, suggesting that they ensemble feature map points. We provide standard deviation of feature maps by block depth with ResNer-50 on CIFAR-100. c1 to c4 and s1 to 24 each stand for stages and spatial smoothing layers, respectively. Model uncertainty is represented by the average standard deviation of several feature maps obtained from multiple NN executions. Data uncertainty is represented by the standard deviation of feature map points obtained from one NN execution.
An unnormalized probability, ranging from 0 to is allowed as the output of Prob. Then, thanks to the linearity integration, we obtain an unnormalized predictive distribution accordingly. Taking this into account, we propose the following Prob:
Prob(z) ReLU o tanh,(2)
where + > 1. We empirically determine r to minimize NLL, metric that measures both accuracy and uncertainty. See Fig. B.3 for more detailed ablation studies. In addition, we expect upper-bounded functions, e.g., ReLU6(22 z) ReLU min(z, 6) and feature map scaling 2/7 with > which BatchNorm, to be able to replace tanh, in Prob; and as expected, these alternatives improve uncertainty estimation in addition accuracy. See Appendix C.2 and Appendix C.3 for detailed discussions on activation (ReLU BatchNorm) and ReLU6 as Prob.
124
126
128
Blur: Averaging neighboring probabilities. Blur averages the probabilities from feature maps. We primarily use the average pool with a kernel size of 2 and stride of as the implementation of Blur the sake of simplicity. Nevertheless, we could generalize Blur by using the following depth-wise convolution, which acts on each input channel separately, with non-trainable kernel
1 [
K
k
(8)
where &s is ID matrix, e.g., k k {(1), (1, (1, 6,,4,6,444 1)). Different ks derive different importances for neighboring feature maps. We empirically show that most Blurs improve the predictive performance and that optimal K varies by model. For more ablation studies, see Table B.2.
130
132
2.2
How DOES SPATIAL SMOOTHING HELP OPTIMIZATION?
133
We present theoretical and empirical aspects to show that spatial smoothing ensembles feature maps.
134
Feature map variance. BNNs have two types of uncertainties: One model uncertainty and the other is data uncertainty (Park et al., 2021). These randomnesses increase the variance of the feature maps. To demonstrate that spatial smoothing is an ensemble, we use the following proposition:
136
137
Proposition Ensembles reduce the variance of predictions.
138
We omit the proof since is straightforward. In our context, predictions are output feature maps of a stage. We investigate model and data uncertainties of the predictions along NN layers to show that spatial smoothing reduces the randomnesses and ensembles feature maps. Figure shows the model
140
uncertainty and data uncertainty of Bayesian ResNet including MC dropout layers. In this figure, the
5

---

Under review as a conference paper at ICLR 2022
1.0
6.0
Log amplitude
Accuracy (%)
70
4.0-
65+
2.0-
0.0m
0.5xt
1.0rc
0.2mt
0.5
0.8x
Frequency
Noise frequency
(a) Frequency mask
(b) Log amplitude at c1
(c) Robustness for noise frequency
Detemainistic
Deterministic
Smooth
MC dropout
MC drepout Smooth
Figure 6: MC dropout adds high-frequency noises, and spatial smoothing filters high-frequency signals. In these experiments, we use ResNet-50 for ImageNet. Left: Frequency mask M, with 0.1m. Middle: Diagonal components of Fourier transformed feature maps at the end of the stage Right: The accuracy against frequency-base -based random noise. ResNets are vulnerable to high-frequency noises. Spatial smoothing improves the robustness against high- feghffreyeeccy noises.
142
uncertainty of MC dropout's feature map only accumulates, and almost monotonically increases in every NN layer. In contrast, the uncertainty of "MC dropout spatial smoothing" feature map is significantly decreases at the end of stages, suggesting that the smoothing layers ensemble the feature map. In other words, they make the feature map more accurate and stabilized input volumes for the next stages. In addition, consistently, the spatial smoothing layer close to the last layer significantly improves performance because it reduces the uncertainty of predictions largely. See Fig. B.5 for more detailed results. Deterministic NNs do not have model uncertainty but data uncertainty. Therefore, spatial smoothing improves the performance of deterministic NNs as well Bayesian NNs.
144
145
148
147
148
149
150
Fourier analysis. We also analyze spatial smoothing through the lens of Fourier transform: Proposition Ensembles filter high-frequency signals.
152
The proof is provided in Eqs. (16) to (17). Figure 6b shows the 2D Fourier transformed output feature map at the end of the stage 1. This figure reveals that MC dropout almost does not affect low- fww-freueenyy (< 0.3r) ranges, and it adds high-frequency (2 0.3m) noises. Since spatial smoothing is a low-pass filter, it effectively filters high-frequency signals, including the noises caused by MC dropout.
153
154
155
156
We also find that CNNs are particularly vulnerable high-frequency noises. To demonstrate this claim, following Shao et al. (2021), we measure accuracy with respect to data with frequency-base random noise Cnoise Io + F-1 (F(6) @ My), where zy is clean data, F()) and F-'(.)) Fourier transform and inverse Fourier transform, is random noise, and M, is frequency mask as shown in Fig. 6a. Figure 6c exhibits the results. In sum, figh-freeuency noises, including those caused by MC dropout, significantly impair accuracy. Spatial smoothing improves the robustness by effectively removing high-frequency noises.
158
160
162
164
Loss landscape. Lastly, show that the randomness hinders NN training as follows:
Proposition 3. Randomness of predictions sharpens the loss landscape, and ensembles flatten it.
The proof is provided in Eqs. (18) (25). Since sharp loss function disturbs NN optimization (Keskar et al., 2017; Santurkar et al., 2018; Foret et al., 2020), reducing the uncertainty helps NN learn strong representations. For example, training phase NN ensemble averages the randomness, and it flattens the loss function. In consequence, an ensemble of BNN outputs in training phase significantly improves the predictive performance. See Fig. D.4 for numerical results. However, we do not use training phase ensemble because it significantly increases the training time. Instead, we use spatial smoothing as method that ensembles feature maps without sacrificing training time.
187
188
189
170
172
173
We visualizes the loss landscapes (Li et al., 2018), the contours of NLL on training dataset. Figure 8b shows that the loss landscapes of MC dropout fluctuate and have irregular surfaces due to the
174

---

Under review as a conference paper at ICLR 2022
(a) MLP classifier
(b) GAP classifier (vanilla)
(c) GAP classifier + Smooth
Figure 8: Both GAP and spatial smoothing smoothen the loss landscapes. To demonstrate this, we present the loss landscape visualizations of ResNet-18 18 models with MC dropout on CIFAR-100.
175
randomness. As Li al. (2018); Foret et (2020) pointed out, this may lead to poor generalization and predictive performance, Spatial smoothing reduces randomness as discussed above, and sputial smoothing aidy in optimization by stabilizing and flattening the loss landscape of BNN as shown in Fig. Sc.
MLL
176
177
GAP Smooth
Density
179
Furthermore, we use Hcssian to quantitatively represent the sharp- ness of the loss landscapes. Figure 7 shows the Hessian max eigen- value spectra of the models in Fig. 8 with batch size of 128, which reveals that spatial smoothing reduces the magnitude of Hessian eigenvalues and suppresses outliers. Since large Hessian eigenval- ues disturb NN training (Ghorbani et al., 2019), we come to the conclusion that spatial smoothing helps NN optimization. See Appendix C. for more detailed description of the configurations of the Hessian max eigenvalue spectra. In addition, from these observations, we propose the conjecture that the flatter the loss landscape, the better the uncertainty estimation, and vice versa.
181
183
500
1000
Max Eigenvlaue
185
Figure Both GAP and spa- tial smoothing suppress large Hessian eigenvalue outliers, i.e., they flatten the loss land- scapes. Compare with Fig. 8.
186
187
188
189
190
2.3
REVISITING GLOBAL AVERAGE POOLING
192
The success of GAP classifier in image classification is indisputable. The initial motivation and the most widely accepted explanation for this success that GAP prevents overfitting by using far fewer parameters than multi-layer perceptron (MLP) (Lin al., 2014). However, we discover that the explanation is poorly supported. We compares GAP with other classifiers including MLP. Contrary popular belief, Table 1 suggests that MLP does not overfit the training dataset. MLP underfits or gives comparable performance to GAP the training dataset. On the test
Table 1: MLP does not overfit the training dataset. We report train- ing NLL (NLL4rana) and testing NLL (NLL of ResNet-50 on CIFAR-100.
193
194
196
197
CLASSIFIER NLLuain NLL
196
GAP
0.0061
0.822
200
MLP
0.0071
1.029
202
dataset, GAP provides better results compared with MI.P. See Table C. for more detailed results.
Our argument is that GAP is extreme of spatial smoothing. In other words, GAP is successful because it ensembles feature maps and smoothens the loss landscape to help optimization. To support this claim, we visualizes the loss landscape of MLP shown in Fig. 8a. It chaotic compared to that of GAP as shown in Fig. 8b. Hessian shows the consistent results as demonstrated by Fig. 7.
204
205
206
207
EXPERIMENTS
This section presents two experiments. The first experiment is image classification through which we show that spatial smoothing not only improves the ensemble efficiency, but also the accuracy, uncertainty, and robustness of both deterministic NN and MC dropout. The second experiment is semantic segmentation on data streams through which we show that spatial smoothing and temporal smoothing (Park et al., 2021) complementary. See Appendix A for more detailed configurations.
209
211
7

---

Under review as a conference paper at ICLR 2022
0.96
NLL
Accuracy
76.5
3.0
ECE
2.5
76.0
0.92
2.0
10
10
10P
10
10
10
Ensemble Size
Ensemble Size
Ensemble Size
0
Deterministic
dropout
dropout Smooth
Figure 9: Spatial smoothing also improves predictive performance on large datasets. We report predictive performance of ResNet-50 on ImageNet.
Three metrics are measured in these experiments: NLL (4'), accuracy (t), and expected calibration 214 enror (ECE, J) (Guo et al., 2017). NLL represents both accuracy and uncertainty, and is the most widely used as proper scoring rule. ECE measures discrepancy between accuracy and confidence.
3.1
IMAGE CLASSIFICATION
217
This section mainly discuss ResNet (He et al., 2016a). Table E.I also discuss other settings that show the same trend: e.g,, VGG (Simonyan & Zisserman, 2015), ResNeXt (Xie al., 2017), and pre-activation models (He et al., 2016a). Spatial smoothing also improves deep ensemble Lakshminaraaanan et al., 2017), another non-Bayesian probabilistic NN method. See Fig. E.I
218
219
220
Performance. Fig. 3 and Fig. 9 show the predictive performances of ResNet-18 CIFAR-100 and ResNet-50 on ImageNet, respectively. The results indicate that spatial smoothing improves both accuracy and uncertainty in many respects. Let us be more specific. First, spatial smoothing improves the efficiency of ensemble size. In these examples, the NLL of "MC dropout spatial smoothing" with ensemble size of 2 is comparable or even better than that of MC dropout with ensemble size of 50. In other words, "MC dropout spatial smoothing" is 25x faster than MC dropout with similar predictive performance. Second, the predictive performance of "MC dropout + spatial smoothing" is better than that of MC dropout, at an ensemble size of 50. Third, spatial smoothing improves the predictive performance of deterministic NN, as well as MC dropout.
222
224
226
227
228
229
To evaluate robustness against data corruption, we measure predictive performance of ResNet- 18 on CIFAR-100- C (Hendrycks & Dietterich, 2019). This dataset consists of data corrupted by 15 different types, each with levels of intensity each. We use mean corruption NLL (mCNLL, 4), the averages of NLL over intensities and corruption types, to summarize the
Robustness.
70%
231
dropont
dropont Smooth
MCNLL (%)
Deterministic
233
Determinstti Smooth
65
235
236
performance of corrupted data in single value. See Eq. (32) for
237
a more rigorous definition. Figure 10 shows that spatial smoothing
60
238
not only improves the efficiency but also corruption robustness across whole range of ensemble size. See Fig. E.3 for more details. Spatial smoothing also improves adversarial robustness and perturbation consistency (f) (Hendrycks & Dietterich, 2019; Zhang, 2019a), 9a),
10a
10 Ensemble Size
239
240
Figure 10: Spatial smoothing improves the robustness. See
242
invariance. See Table E.2, Table E.3, and Fig, E.4 for more details.
Fig. E.3 for more details.
3.2
SEMANTIC SEGMENTATION
245
Table 2 summarizes the result of semantic segmentation on Cam Vid dataset (Brostow et al., 2008) that consists of real-world 380x480 pixels videos. The table shows that spatial smoothing improves predictive performance, which is consistent with the image classification experiment. Moreover, the result reveals that spatial smoothing and temporal smoothing (Park al., 2021) are complementary. See Table E.4 for more results.
247
249
We arrows indicate which direction is better.
8

---

Under review as a conference paper at ICLR 2022
Table Spatial smoothing and temporal smoothing are complementary. We provide predictive performance of MC dropout in semantic segmentation. SPAT and TEMP each stand for spatial smoothing and temporal smoothing. Acc and CONS stand for accuracy and consistency. The numbers in brackets denote the performance improvements over the baseline.
AcC (%)
ECE (%)
CoNS (%)
SPAT
TEMP
NLL
0.298 (-0.000)
92.5 (+0.0)
4.20 (-0.00)
95.4 (+0.0)
0.284 (-0.014)
92.6 (+0.1)
3.96 (-0.24)
95.6 (+0.2)
v
0.273 (-0.025)
92.6 (+0.1)
3.23 (-0.97)
96.4 (+1.0)
0.260 (-0.038)
92.6 (+0.1)
2.71 (-1.49)
96.5 (+1.1)
250
4
RELATED WORK
251
Spatial smoothing can be compared with prior works in the following areas.
252
Anti-aliased CNNs. Local means (Zhang, 2019a; Zou et al., 2020; Vasconcelos et al., 2020; Sinha al., 2020) were introduced for the shift-invariance of deterministic CNNs in image classification. They were motivated to prevent the aliasing effect of subsampling. Although the local filtering can result loss of information, Zhang (2019a) experimentally observed an increase in accuracy that
253
254
255
was beyond expectation. We provide fundamental explanation for this phenomenon: Local means ensemble not only improves accuracy, but also uncertainty and robustness
257
are a spatial ensemble. of deterministic and Bayesian NNs. In Fig. F.I. also show that the predictive performance improvement is not due to anti- anti-aliasing of local mean. See Appendix for more discussion on local means. For a discussion on non-local means (Wang et al., 2018) and self-attention (Dosovitskiy et al., 2021) 1), see Section 5.
259
261
262
Sampling-free BNNs. Sampling-free BNNs (Hernández-Lobatt & Adams, 2015; Wang et al., 2016; Wu et al., 2019) predict results based on single couple of NN executions. To this end, it is assumed that posterior and feature maps follow Gaussian distributions. However, the discrepancy
283
254
285
between reality and assumption accumulates in every NN layer. Consequently, to the best of our knowledge, most of the sampling free BNNs could only be applied to shallow models, such as LeNet, and were tested on small datasets. Postels et al. (2019) applied sampling-free BNNs to SegNet; nonetheless, Park et al. (2021) argued that they do not predict well-calibrated results.
286
267
Efficient deep ensembles. Deep ensemble (Lakshminarayanann et al., 2017; Fort et al., 2019) is another probabilistic NN approach for predicting reliable results. BatchEnsemble (Wen et al., 2020; Dusenberry et al., 2020) ensembles over low-rank subspace to make deep ensemble more efficient. Depth uncertainty network (Antoran et al., 2020) aggregates feature maps from different depths of single NN predict results efficiently. Despite being robust against data corruption, it provides weaker predictive performance compared to deterministic NN and MC dropout.
270
272
274
275
DISCUSSION
278
We propose spatial smoothing, a simple yet efficient module to improve BNN. Three different per- spectives, namely, feature map variance, Fourier analysis. and loss landscape, suggest that spatial smoothing ensembles feature maps. The limitation of spatial smoothing that designing its compo-
277
278
279
nents requires inductive bias. In other words, the optimal shape of the blur kernel is model--depnndent.
280
We believe this problem can be solved by introducing self-attention (Vaswani et al., 2017). Self- attentions for computer vision (Dosovitskiy et al., 2021; Touvron et al., 2021; Carion et al., 2020)
282
can be deemed as trainable importance- spportance-weighted ensembles of feature maps. The observation that Transformers are more robust than expected (Bhojanapalli et al., 2021; Shao et al., 2021) supports this
284
claim. Therefore, using self- attentions to generalize spatial smoothing would be a promising future
work because it not only expands work, but also helps deepen understanding of self-attention.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
287
To ensure reproducibility, we provide comprehensive resources, such as code and experimental details. The codebase will be released as open source under the Apache License 2.0. See the supplemental material for the code. Appendix A provides the specifications of all models used in this work. Detailed experimental setup including hyperparameters and ablation study are also available Appendix and Appendix B. De-facto image datasets are used for all experiments as described in Appendix A.
286
289
290
292
REFERENCES
293
Javier Antoran, James Allingham, and José Miguel Hernández-Lobato. Depth uncertainty in neural networks. Advances in Neural Information Processing Systems, 2020.
294
295
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? Journal of Machine Learning Research, 2019.
296
297
Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, and Andreas Veit. Understanding robustness of transformers for image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
298
289
300
Gabriel J Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and recog- nition using structure from motion point clouds. In European Conference on Computer Vision Springer, 2008
301
302
303
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision. Springer, 2020.
304
305
306
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
307
309
Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snock, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1 factors. In International Conference on Machine Learning. PMLR, 2020.
311
313
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry, Exploring the landscape of spatial robustness. In International Conference Machine Learning. PMLR, 2019.
315
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awar aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2020.
317
319
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspec- tive. arXiv preprint arXiv: 2019.
321
Jonathan Frankle, David Schwab, and Ari S Morcos. Training batchnorm and only batchnorm: On the expressive power of random features in enns. In International Conference on Learning Representations, 2021.
323
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning. PMLR, 2016.
325
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
327
328
329
Behrooz Ghorbani. Shankar Krishman, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In International Conference Machine Learning. PMLR, 2019.
330
10

---

Under review as a conference paper at ICLR 2022
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Representations, 2015.
332
333
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate large minibatch sgd: Training imagenet in hour. arXiv preprint 1rXivv:706.02880
334
335
2017.
336
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning. PMLR, 2017.
337
338
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
339
340
2016a.
341
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision. Springer, 2016b.
342
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
344
345
José Miguel Hemmnndez-Lobto and Ryan Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In International Conference on Machine Learning. PMLR, 2015.
346
347
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2020.
349
350
A Kendall, Badrinarayanan, and R Cipolla. Bayesian segnet: Model uncertainty in deep convolu- tional encoder-decoder architectures for scene understanding. In BMVC, 2017.
352
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, 2017.
353
354
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017.
355
358
Alex Krizhevsky and Geoff Hinton. Convolutional deep belief networks on cifar- 10. Unpublished manuscript, 2010.
358
359
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
360
Balaji Lakshminarayanann, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing
381
362
Systems, 2017.
383
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems, 2018.
354
365
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In International Conference on Learning Representations, 2014.
366
Antonio Loquercio, Mattia Segu, and Davide Scaramuzza. A general framework for uncertainty estimation in deep learning. IEEE Robotics and Automation Letters, 2020
368
389
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
370
371
372 A Malinin and M Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2018.
373
374
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, 2019.
376
377
11

---

Under review as a conference paper at ICLR 2022
Namuk Park, Taekyu Lee, and Songkuk Kim. Vector quantized bayesian neural network inference for data streams. In AAAI Conference on Artificial Intelligence, 2021.
379
380
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 2019.
381
382
383
384
Janis Postels, Francesco Ferroni, Huseyin Coskun, Nassir Navab, and Federico Tombari. Sampling- free epistemic uncertainty estimation using approximated variance propagation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
385
386
387
Olaf Ronneberger, Philipp
Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention. Springer, 2015.
389
390
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, pp. 211-25 2015.
391
392
393
394
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
395
396
397
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? Advances in Neural Information Processing Systems, 2018.
398
389
Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin-Yu Chen, and Cho-Jui Hsieh. On the adversarial robustness of visual transformers. arXiv preprini arXiv:2 15670, 2021.
400
401
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for scalesscale image recognition. In International Conference on Learning Representations, 2015.
402
403
Samarth Sinha, Animesh Garg, and Hugo Larochelle. Curriculum by smoothing. Advances in Neural Information Processing Systems, 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning. PMLR, 2021.
406
408
Cristina Vasconcelos, Hugo Larochelle, Vincent Dumoulin, Nicolas Le Roux, and Ross Goroshin. An effective anti-aliasing approach for residual networks. arXiv preprint arXiv:2011.10675 2020.
409
410
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
411
Hao Wang, Xingjian Shi, and Dit- Yan Yeung. Natural- -aramme- networks: A class of probabilistic neural networks. Advances in Neural Information Processing Systems, 2016.
414
415
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
416
417
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations, 2020.
418
Anqi Wu, Sebastian Nowozin, Edward Meeds, Richard E Turner, José Miguel Hernández-LLobaoo and Alexander Gaunt. Deterministic variational inference for robust bayesian neural networks. In International Conference on Learning Representations, 2019.
420
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEÉE Conference on Computer Vision and Pattern Recognition, 2017.
423
12

---

Under review as a conference paper at ICLR 2022
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. 2020 IEEE International Conference on Big Data (Big Data). IEEE, 2020.
426
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Advances in Neural Information Processing Systems, 2018.
429
430
431
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
432
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning. PMLR, 2019.
434
435
Richard Zhang. Making convolutional networks iiftt-invariant again. In International Conference on Machine Learning. PMLR, 2019a.
436
437
Richard Zhang. Official meta-review of making convolutional networks shift-invariant again, 2019b. URL httpss///ont //tppr/ee///eeriieeeeee http://rruu.........aaaaneettte
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
440
Xueyan Zou, Fanyi Xiao, Zhiding Yu, and Yong Jae Lee. Delving deeper into anti-aliasing in convnets. In BMVC, 2020.
443
13

---

Under review as a conference paper at ICLR 2022
1.1
78
Accuracy
BCE 550
7.5
1.0
76
5.0
74-
2.5
0.8
20
50
10
20
40
50
MC dropout rate (%)
MC dropout rate (%)
MC dropout rate (%)
drapout (N-1)
drapout (N-50)
drapout (N-1)
dropout (N-50)
Figure A.I: Spatial smoothing improves predictive performance at all dropout rates, As the dropout rate increases, both accuracy and ECE decrease. The performance is optimized when aceuracy and uncertainty are balanced.
A EXPERIMENTAL SETUP AND DATASETS
445 We obtain the main experimental results with the Intel Xeon W-2123 Processor, 32GB memory, and single GeForce RTX 2080 Ti for CIFAR (Krizhevsky et al., 2009) and Cam Vid (Brostow et al., 2008). For ImageNet (Russakovsky et al., 2015), we use AMD Ryzen Threadripper 3960X 24-Core Processor, 256GB memory, and four GeForce RTX 2080 Ti. We conduct ablation studies with four
446
447
449
Intel Intel Broadwell CPUs, 15GB memory, and single NVIDIA T4. Models are implemented in PyTorchPPaszke et al., 2019). The detailed configurations of image classification and semantic segmentation are as follows.
451
452
A.1
IMAGE CLASSIFICATION
453
We use VGG (Simonyan & Zisserman, 2015), ResNet (He et al., 2016a), pre-activation ResNet (He al.. 201 2a), and ResNeXt (Xie al., 2017) in image classification. According the structure suggested by Zagoruyko & Komodakis (2016), each block of Bayesian NNs contains one MC dropout layer.
454
455
456
457
NNs are trained using categorical cross-entropy loss and SGD optimizer with initial lcarning rate of 0.1. 1, momentum 0.9, and weight decay of x 10-4, We also use multi-step learning rate scheduler with milestones at 60, 130, and 160, and gamma of 0.2 on CIFAR, and with milestones at 30, 60,
458
459
460
and 80, and gamma of 0.2 on ImageNet. We train NNs for 200 epochs with batch size of 128 on CIFAR, and for 90 epochs with batch size of 256 on ImageNet. We start training with gradual warmup Goyal et al., 2017) for epoch on CIFAR. Basic data augmentations, namely random cropping and used. One exception is the training of ResNeXt ImageNet. In this case, use the batch size of 128 and learning rate of 0.05 because of memory limitation.
462
horizontal flipping,
464
We hyperparameters that minimizes NLL of ResNet. Table A.1 provides hyperparameters for deterministic and Bayesian NNs. For fair comparison, models with and without spatial smoothing share hyperparameters such as MC dropout rate. However, Fig. A. shows that spatial smoothing improves predictive performance of ResNet-18 at all dropout rates on CIFAR-100. The default ensemble size of MC dropout is 50. We report averages of three evaluations, and error bars in figures represent min and max values. Standard deviations are omitted from tables for better visualization. See the source code released on GitHub for other details.
466
468
470
471
472
A.2
SEMANTIC SEGMENTATION
473 We use U-Net (Ronneberger et al., 2015) in semantic segmentation. Following Bayesian SegNet (Kendall et al., 2017), Bayesian U-Net contains six MC dropout layers. We add spatial smoothing before each subsampling layer in U-Net encoder. We use previous predictions and decay rate of -08 for temporal smoothing.
475
14

---

Under review as a conference paper at ICLR 2022
Table A.I: Hyperparameters of models for image classification.
MC DROPOUT RATE
DATASET
MODEL
TEMPERATURE
(%)
30
VGG
2
10
30
2
10
30
ResNet
2
10
CIFAR-10 CIFAR- 100
30
2
10
30
Preact-Reshhe
2
10
30
2
10
30
ResNeXt
2
10
30
2
10
5
ResNet
2
10
5
2
10
ImageNet
5
ResNeXt
10
5
2
10
477 Cam Vid consists 760x960 pixels road scene video sequences. We resize the image bilinearly to
360 480 pixels. We use list reduced to 11 labels by following previous works, e.g. (Kendall & Gal, 2017).
479
NNs are trained using categorical cross-entropy loss and Adam optimizer with initial learning rate of 0.001 and 31 of 0.9, and B2 of 0.999. We train NN for 130 epoch with batch size of 3. The learning rate decreases to 0.0002 the 100 epoch. Random cropping and horizontal flipping used for data augmentation. Median frequency balancing is used to mitigate dataset imbalance. Other details follow Park al. (2021).
481
483
485
B
ABLATION STUDY
The probabilistic spatial smoothing proposed in this paper consists of two components: Prob and Blur. This section explores several candidates for each component and their properties.
10
487
ReLU
486
tanh
ReLUG
Cons Constant
490
B. PROB: FEATURE MAPS TO PROBABILITIES
BatchNorm
491
define Prob as a composition of an upper- bounded function and ReLU, function that imposes the lower bound zero. Fig. B.I shows widely used upper-bounded functions: tanh-(x:)
493
tanh(x// ( ReLUs(x: = min(max(2, 6), 0), and constant scaling which is z/T.
0-
495
0.0
2.5
5.0
7.5
10.0
Figure B.I: Upper-bounded functions as candidates of Prob.
497
Table B. shows the predictive performance improve- ment by Prob with various upper-hounded functions
15

---

Under review as a conference paper at ICLR 2022
Table B.1: We use tanh the default for Prob based on the predictive performance of MC dropout for CIFAR-100 100 with various Probs.
ACC (%)
ECE (%)
MODEL
SMOOTH
NLL
1.133 (-0.000)
68.8 (+0.0)
3.66 (+0.00)
ReLU tanh
1.064 (-0.069)
70.4 (+1.6)
2.99 (-0.67)
ReLU ReLU6
1.093 (-0.040)
69.8 (+1.0)
4.26 (+0.60)
ReLU Constant
0.995 (-0.138)
72.5 (+3.7)
2.11 (-1.55)
VGG-16
Blur
0.985 (-0.000)
72.4 (+0.0)
1.77 (+0.00)
Blur ReLU tanh
0.984 (-0.001)
72.7 (+0.3)
2.07 (+0.30)
Blur ReLU ReLU6
0.982 (-0.003)
72.5 (+0.1)
1.84 (+0.07)
Blur ReLU Constant
0,991 (+0.005)
72.9 (+0.5)
1.03 (-0.74)
1.215 (-0.000)
67.3 (+0.0)
6.37 (+0.00)
ReLU tanh
1.131 (-0.084)
69.2 (+1.9)
5.23 (-1.14)
ReLU ReLU6
1.166 (-0.049)
68.3 (+1.0)
6.44 (-0.06)
ReLU Constant
0.997 (-0.218)
72.5 (45.2)
1.09 (-5.29)
VGG-19
Blur
1.039 (-0.000)
71.1 (+0.0)
3.12 (+0.00)
Blur ReLU tanh
1.034 (-0.005)
71.3 (40.2)
3.31 (+0.19)
Blur ReLU o ReLU6
1.038 (-0.002)
71.3 (+0.2)
3.84 (+0.72)
Blur ReLU Constant
0.995 (-0.045)
72.3 (+1.2)
1.41 (-1.71)
0.848 (-0.000)
77.3 (+0.0)
3.01 (+0.00)
ReLU tanh
0.838 (-0.010)
77.7 (+0.4)
2.92 (-0.08)
ReLU ReLU6
0.844 (-0.004)
77.4 (+0.1)
2.74 (-0.27)
ReLU Constant
0.825 (-0.023)
77.7 (40.4)
1.87 (-1.14)
ResNet-18
Blur
0.806 (-0.000)
78.6 (+0.0)
2.56 (+0.00)
Blur o ReLU o tanh
0.801 (-0.005)
78.9 (+0.3)
2.56 (-0.01)
Blur ReLU ReLU6
0.805 (-0.001)
78.9 (+0.2)
2.59 (+0.03)
Blur ReLU Constant
0.811 (+0.005)
78.5 (-0.2)
1.84 (-0.72)
0.822 (-0.000)
79.1 (+0.0)
6.63 (+0.00)
ReLU tanh
0.812 (-0.010)
79.3 (+0.2)
6.74 (+0.11)
ReLU ReLU6
0.799 (-0.023)
79.4 (+0.3)
6.71 (+0.08)
ReLU Constant
0.788 (-0.034)
79.6 (+0.5)
5.22 (-1.41)
ResNet-50
Blur
0.798 (-0.000)
80.0 (+0.0)
7.21 (+0.00)
Blur ReLU o tanh
0,800 (+0.002)
80.1 (+0.1)
7.25 (+0.04)
Blur o ReLU o ReLU6
0.800 (+0.002)
80.2 (+0.2)
7.30 (+0.09)
Blur ReLU Constant
0.779 (-0.019)
80.4 (+0.4)
5.81 (-1.40)
16

---

Under review as a conference paper at ICLR 2022
10.0
1.00
8.0
0.75
6.0
0.50
4.0
2.0
0.25
0.00
0.0
2.5
5.0
7.5
10.0
0.0
2.5
5.0
7.5
10.0
Rel.U.
T
0.5
T=1
t=2
T=
T=
Figure B.2: Temperaturr--saledd tanhs (left) and their first derivatives (right) for different tem- peratures.
79-
6.04
1.0
78-
5.0
Accuracy
77-
ECE
4.0
0.9
76-
3.0
0.8
10a
10
10a
10a
10a
10a
Temperature
Temperature
Temperature
dropout Smoooh (N-1)
dropout (N-50)
Figure B.3: The temperature controls the trade-off between accuracy and uncertainty. The accuracy increases the temperature increases, but predictions become more overconfident.
CIFAR-100. In this experiment, we use models with MC dropout, and r = 5 for constant scaling. The results indicate that upper-bounded functions with ReLU tend to improve accuracy 500
uncertainty at the same time. In addition, they show that Prob and Blur are complementary. The best results are obtained when using both Prob and Blur. For the main experiments, we use the composition of tanh, and ReLU Prob, because the hyperparameter of constant scaling is highly dependent dataset and model.
502
504
505
Temperature. The characteristics of temperature-scaled tanh depends on т. Figure B.2 plots tanh, and their first derivatives with various temperatures. shown in this figure, tanh, has couple of useful properties. First, tanh, has an upper bound of r. Second, the first derivative of tanh, at does not depend on T.
506
507
508
509
Fig. B.3 shows the predictive performance of ResNet-18 with MC dropout and spatial smoothing for the temperature on CIFAR-100. this figure, the accuracy increases the temperature increases. In terms of ECE, NN predicts more underconfident results as 7 decreases. It is misinterpretation that the result overconfident at low 7 because ECE high. By definition, ECE relies on the absolute value of the difference between confidence and accuracy. In this example, at low т, the accuracy is greater than the confidence, which leads to a high ECE. Moreover, at t = 0.2, ECE with = 50 is greater than that with N 1. which means that the result is severely underconfident. NLL, metric representing both accuracy and uncertainty, is minimized when the accuracy and the uncertainty are balanced. In conclusion, set the default value of 10.
510
512
514
516
17

---

Under review as a conference paper at ICLR 2022
0.00
0.02
0.02
0.02
0.00
0.02
0.06
0.09
0.06
0.02
0.06
0.12
0.06
0.02
0.09
0.14
0.09
0.02
0.25
0.25
0.12
0.25
0.12
0.06
0.09
0.06
0.02
1.00
0.25
0.25
0.06
0.12
0.06
0.00
0.02
0.02
0.02
0.00
(a) k = (1)
(b) k = 11
(c) k =
11
(d) k = (1, 4,4,,
Figure B.4: Kernels for Blur. Brighter background indicates higher importance.
Table B.2: The optimal shape of the blur kernel is model-dependent. We measure the predictive performance of MC dropout using spatial smoothing with various size of Blur kernels on CIFAR-100 100.
AcC (%)
ECE (%)
MODEL
[
NLL
1.087 (-0.000)
69.8 (+0.0)
3.43 (-0.00)
1.034 (-0.053)
71.4 (+1.6)
1.06 (-2.37)
VGG-16
0.986 (-0.101)
72.7 (+2.9)
1.03 (-2.40)
1.018 (-0.069)
72.0 (42.2)
1.32 (-2.11)
1.096 (-0.000)
69.8 (+0.0)
4,74 (-0.00)
2
1.071 (-0.025)
70.4 (+0.6)
2.15 (-2.59)
VGG-19
3
1.026 (-0.070)
71.9 (+2.1)
2.56 (-2.18)
5
1.032 (-0.064)
71.6 (+1.8)
2.16 (-2.58)
0.840 (-0.000)
77.6 (+0.0)
2.63 (-0.00)
0.801 (-0.039)
78.9 (+1.4)
2.56 (-0.07)
ResNet-18
0.822 (-0.018)
78.7 (+1.1)
2.86 (-0.23)
0.837 (-0.003)
78.4 (+0.8)
3.05 (-0.42)
0.814 (-0.000)
79.5 (+0.0)
6.56 (-0.00)
0.806 (-0.008)
80.0 (+0.5)
7.35 (+0.79)
ResNet-50
0.796 (-0.019)
79,9 (+0.4)
7.38 (+0.82)
0.816 (+0.001)
79.4 (-0.1)
7.38 (40.82)
18

---

Under review as a conference paper at ICLR 2022
8.0
78-
1.00
0.95
Accuracy
77-
BCE. (%)
6.0
0.90
76-
4.0
0.85
7
2.0
10
Ensemble Size
Ensemble Size
Ensemble Size
None
81
a2
83
24
Figure B.5: Spatial smoothing close to the last layer (s3) significantly improves performance. We report predictive performance of ResNet-18 with spatial smoothing after each stage CIFAR-106 100. None indicates vanilla MC dropout.
518
B.2
BLUB: AVERAGING NEIGHBORING PROBABILITIES
519
Blur is depth-wiee wise convolution with kernel. The kernel given by Eq. (8) derived from various ks such as & E {(1), (1, 1), (), (1,4, 6,4, 1)}. In these examples, if k is 1, Blur is identity. k/ 2, Blur box blur, which is used in the main experiments. If k is 5, Blur is an approximated Gaussian blur.
521
523
Table B.2 shows predictive performance of models using spatial smoothing with kernels on CIFAR-100. This results show that most kernels improve both accuracy and uncertainty. However, most effective kernel size depends on the model.
525
526
B.3
POSITION OF SPATIAL SMOOTHING.
527
As shown in Fig. 5, the magnitude uncertainty tends to increase as the depth increases. Therefore, expect that spatial smoothing close the output layer will mainly drive performance improvement.
529
We investigate the predictive performance models with MC dropout using only one spatial smooth- ing layer. Figure B.5 shows the predictive performance of ResNet-18 with spatial smoothing after each stage on CIFAR-100, The results suggest that spatial smoothing after s3 is the most important for improving performance. Surprisingly, spatial smoothing after s4 the least important. This is because GAP, the most extreme case of spatial smoothing, already exists there.
532
C REVISITING PRIOR WORKS
534
535
As mentioned in Section prior works-namehh GAP, pre-activation, and ReLU6-are spacial cases of spatial smoothing. This section discusses them in detail.
536
C.1 GLOBAL AVERAGE POOLING
The composition of GAP and fully connected layer is the most popular classifier in classification tasks. The original motivation and the most widely accepted explanation for the success is that GAP classifier prevents overfitting because it uses significantly fewer parameters than MLP (Lin et al., 2014). To verify this claim, we measure the predictive performance MLP, GAP, and global max pooling (GMaxP), classifier that uses the number of parameters GAP, training dataset.
539
541
Predictive performance. Table C.I shows the experimental results on the training and the test dataset of 100, suggesting that the explanation is poorly supported. On both the training and the test dataset, most predictive performance of MLP is worse than that of GAP. It is counter- intuitive result meaning that MLP do not overfit the training dataset. In addition, the performance improvement by GAP is remarkable in VGG, which has irregular loss landscape. The predictive
544
546
19

---

Under review as a conference paper at ICLR 2022
Table C.I: MLP classifier does not overfit training dataset, i.e., GAP does not regularize NNs. We
provide predictive performance of MC dropout with various classifiers on CIFAR-100. ERR is error
TRAIN
TEST
MODEL
CLASSIFIER
ERR (%)
ECE
AcC (%)
ECE (%)
NLL
NLL
(%)
GAP
0.0852
0.461
6.75
1.030
72.3
3.24
MLP
0.5492
13.1
13.8
1.133
68.8
3.66
VGG-16
GMaxP
0.0846
0.470
6.67
1,050
72.2
3.60
GMedP
0.0867
0.501
6.80
1.042
72.2
3.35
GAP
0.1825
2.50
10.4
1.035
71.9
1,46
MLP
0.7144
17.7
14.8
1.215
67.3
6.37
VGG-19
GMaxP
0.1939
2.85
10.6
1.063
2.10
GMedP
0.1938
2.80
10.6
1,051
71.7
1.70
GAP
0.0124
0.0287
1.19
0.841
77.5
2.92
MLP
0.0076
0.0347
7.22
1.040
74.8
9.55
ResNet-18
GMaxP
0.0113
0.0233
1.41
0.905
76.3
5.23
GMedP
0.0156
0.0347
1.46
0.889
76.4
5.03
GAP
0.0061
0.0220
0.48
0.822
79.1
6.63
MLP
0.0071
0.0370
8.53
1.029
76.9
11.8
Re
GMaxP
0.0074
0.0313
1.09
0.887
77.2
5.67
GMedP
0.0053
0.0287
0.47
0.849
78.5
6.29
10
75
8-
Accuracy (%)
ECE (%)
50
NLL
40
4-
25
20
2-
Test
Test
Test
Intensity
Intensity
Intensity
GMedP
100
92-
100
mCNLL (%)
mCE (%))
90
MCRCE
80
80
88
60
60
86
40
109
10
109
10
10"
10a
Ensemble Size
Ensemble Size
Ensemble Size
GAP
MLP
GMaxP
GMedP
Figure C.I: GAP classifier improves not only the predictive performance on clean dataset but also the robustness. We measure the predictive performance of ResNet- 18 using MC dropout with classifiers on CIFAR-100-C.
20

---

Under review as a conference paper at ICLR 2022
MLP classifier
(b) GAP classifier
(c) GAP classifier + Smooth
Figure C.2: GAP and spatial smoothing flatten the loss landscapes. We visualize the loss landscape sequences ResNet-18 with MC dropout on CIFAR- 100. Although each sequence shares the bases, it fluctuates due to the randomness of the MC dropout.
548
performance of GMaxP is better than that of MLP, but worse than that of GAP. This shows that using fewer parameters partially helps to improve predictive performance; however, it is insufficient to explain the predictive performance improvement by GAP. Finally, global median pooling (GMedP) provides better predictive performance than GMaxP. It implies that using other noise reduction methods instead of average pooling helps to improve predictive performance.
549
551
553
Robustness. To evaluate the robustness of the classifiers, we measure the predictive performance of ResNet-18 using MC dropout with the classifiers CIFAR- 1 Figure C.I shows the experimental results. This figure suggests that ML.P not robust against data corruption, as we would expect. In terms of accuracy, the robustness of GMaxP and GMedP is relatively comparable that of GAP; however, in terms of uncertainty, GAP is the most robust. These are consistent results with other spatial smoothing experiments.
555
557
558
559
Loss landscape visualization. To understand the mechanism of GAP performance improvement, investigate the loss landscape. Figure C.2 shows the loss landscape sequences of ResNet with MC dropout. In this figure, each sequence shares the bases, but they fluctuate due to the randomness of the MC dropout. Figure C.2a is the loss landscape of the model using MLP classifier instead of GAP classifier. The loss landscape is chaotic and irregular, resulting in hindering and destabilizing NN optimization. Fig. C.2b loss landscape sequence of ResNet with GAP classifier. Since GAP ensembles all of the feature map points at the last stage, flattens and stabilizes the loss landscape. Likewise, as shown in Fig. C.2c, spatial smoothing layers at the end of all stages also flattens and stabilizes the loss landscape.
562
564
566
568
Hessian eigenvalue spectra. To evaluate the smoothness of the loss landscapes quantitatively, also investigate their Hessians at the optimized weights. In particular, calculate Hessian eigenvalue spectra (Ghorbani et al., 2019), distributions Hessian eigenvalues, to show how spatial smoothing helps NN optimization. To this end, try stochastic Lanczos quadrature algorithm
570
21

---

Under review as a conference paper at ICLR 2022
572
implemented by Yao al. (2020). However, the problem is that the model with MLP classifier requires a lot of memory while the algorithm is memory inefficient.
573
In the training phase, we calculate the mean gradients with respect to mini- rather than the entire dataset. Therefore, it may be reasonable to investigate the properties of the Hessian "mini- batch-wisely". For that purpose, we propose method, Hessian max eigenvalue spectra, that evaluates the distribution of "Hessian's 's maximum eigenvalues for one mini-batch". We use power iteration to produce only the greatest eigenvalue of the Hessian. This algorithm is easy implement and requires significantly less memory and computational cost, compared with stochastic Lanczos quadrature with respect to entire dataset. With this method, we can investigate the Hessian of NNs with MLP classifiers, which would require a lot of GPU memory.
575
576
577
578
579
580
581
582
Figure 7 shows the Hessian max eigenvalue spectra of GAP classi- fier models with and without spatial smoothing layers. As Li et al. (2018); Foret et al. (2020) and Appendix D.3 pointed out, Hessian eigenvalue outliers disturb NN training. This figure explicitly show that the GAP and spatial smoothing reduce the magnitude of the Hessian eigenvalues and suppress the outliers, which leads to the
583
Smooth
584
Log density
586
588
same result as the previous visualizations: GAP as well as spatial smoothing smoothen the loss landscape. In conclusion, avverag- ing feature map points tends to help neural network optimization by smoothing, flattening, and stabilizing the loss landscape. We observe a similar phenomenon for deterministic NNs. We also evaluate the Hesse eigenvalue spectrum as shown in Fig. C.3, and it leads to the same conclusion.
590
mww A A
20
582
Eigenvlaue
593
594
Figure C.3: Spatial smoothing suppress eigenvalue outliers. We provide Hessian eigenvalue spectra of ResNet- 18 with MC dropout on CIFAR-100. See also Fig. 7.
595
In these experiments, we use MLP incorporating dropout layers with a rate of 50% as the classifier. Since the dropout is one of the factors that makes MLP underfit the training dataset, we also evaluate MLP using dropouts with a rate of 0%. Nevertheless, the results still shows that the predictive performance of MLP is worse
596
597
598
599
than that of GAP on the training dataset. Moreover, it severely degrades predictive performance of ResNet on the test dataset.
601
602
C.2 PRE-ACTIVATION
603
He et al. (2016b) experimentally showed that the pre-activation arrangement, in which the activation ReLU BatchNorm is placed before the convolution, improves the accuracy of ResNet. Since ys of most BatchNorms in CNNs are near-zero (Frankle et al., 2021), BatchNorms reduce the magnitude of feature maps. As shown in Fig. 1.1, constant scaling is non-trainable BatchNorm with no bias, and it also reduces the magnitude of feature map. In Table B.1, we show that constant scaling improves predictive performance. Considering the similarity between Prob with constant scaling and conventional activation, i.e., the similarity between ReLUcConstantSooooommm ing and ReLUoBatchNorrm
605
606
607
608
609
610
we find that the pre-activation arrangement improves uncertainty as well as accuracy, because convolutions act as Blur
611
612
To show this, we change the post-activation of all layers to pre- activatin,,n and measure the predictive performance. For ResNet, we follow the original paper by He et al. (2016b). Table C.2 shows the predictive performance of models with pre-activation. The results suggests that pre-activation improves both accuracy and uncertainty in most cases. For deterministic VGG-19, pre-activation significantly degrades accuracy but improves NLL. In conclusion, they imply that pre-activation is a special case of spatial smoothing.
613
614
615
616
Santurkar et al. (2018) argued that BatchNorm helps in optimization by flattening the loss landscape. 618
We show that spatial smoothing flattens and smoothens the loss landscape, which is consistent explanation. It will be interesting to investigate if BatchNorm helps in ensembling feature maps.
620
C.3
RELU6
622
ReLU6 was experimentally introduced to improve predictive performance (Krizhevsky & Hinton, 2010). Sandler et al. (2018) used "ReLU6 the non linearity because of its robustness when used
623
22

---

Under review as a conference paper at ICLR 2022
Table C.2: Pre-activation arrangement improves uncertainty as well as accuracy. We measure the predietive performance of models with pre-activation arrangement on CIFAR-100.
ACC (%)
ECE (%)
MODEL
MC DROPOUT
PRE-ACT
NLL
2.047 (-0.000)
71.6 (+0.0)
19.2 (-0.0)
1.827 (-0.219)
72.5 (40.9)
19.8 (+0.6)
VGG-16
1.133 (-0.000)
68.8 (+0.0)
3.66 (-0.00)
1.036 (-0.096)
71.7 (+2.9)
3.55 (-0.11)
2.016 (-0.000))
67.6 (+0.0)
21.2 (-0.0)
(-0.217)
64.4 (-3.2)
17.2 (-4.0)
VGG-19
1.215 (-0.000)
67.3 (40.0)
6.37 (-0.00)
1.084 (-0.131)
70.1 (+3.7)
4.23 (-2.14)
0.983 (-0.000)
77.1 (+0.0)
7.75 (-0.00)
0.934 (-0.049)
77.6 (+0.5)
8.04 (+0.29)
ResNet-18
0.937 (-0.000)
76.9 (+0.0)
5.11 (-0.00)
0.872 (-0.065)
77.6 (+0.7)
5.53 (+0.42)
0.880 (-0.000)
79.0 (+0.0)
8.35 (-0.00)
0.870 (-0.010)
79.4 (40.4)
8.27 (-0.08)
ResNet-50
0.831 (-0.000)
78.6 (+0.0)
6.06 (-0.00)
0.819 (-0.012)
79.5 (+0.9)
6.29 (+0.23)
624
with low-precision computation". In Table B.1, we show that ReLU6s at the end of stages helps to ensemble spatial information by transforming the feature map Bernoulli distributions. Since spatial smoothing improves robustness against data corruption, it seems reasonable that ReLU6 is robust to loweprecnnion computation. more abundant investigation into this topic is promising future works.
626
627
628
We measure the predictive performance of NNs using all activations as ReLU6 instead of ReLU. However, in contrast the results in Table B. the results are consistent. We speculate that the reason is that a lot of ReLU6s overly regularize NNs.
629
D EXTENDED ANALYSIS OF HOW SPATIAL SMOOTHING WORKS
631
632
This section provides further explanation of the analysis in Section 2.2.
635
D.
NEIGHBORING FEATURE MAPS IN CNNS ARE SIMILAR
This work exploits the spatial consistency of feature maps, i.e., neighboring feature maps in CNNs 635 are similar. Below, we theoretically and empirically prove the spatial consistency. Moreover, this spatial consistency of feature maps holds even if the input data is spatially inconsistent.
636
Consider single-layer convolutional neural network with one channel:
[w x)
(9)
W
(10)
23

---

Under review as a conference paper at ICLR 2022
(a) single-layer CNN
(b) five-layer CNN with ReLU
Figure D.I: Neighboring feature map points in CNNs are similar, even if input values are iid. provide covariances feature map points with respect to the center feature map (in the red square). Input values Gaussian random noise. Left: A single convolutional layer correlates the target feature map with another feature map that is pixels away, since the kernel size is x.3. Right: A deep CNN more strongly correlates neighboring feature maps.
where is convolution operator with kernel of size k, y is feature map output, uo is kernel weight, and x is input random variable. Then, the covariance of two neighboring feature maps is:
k
Cov(yi, B1+1) = Cov
W 1+1
Wm Wm Um-i)m+22
(11)
m==
Wp (ov 81-22
(12)
k-1
wiit41 of (xx-1+2) +...
(13)
637 where (24-111)) is the variance of 14-141. Therefore, Cov(y 31+1) is non-zero for randomly initialized weights. If is rid, i.e., Cov(2,, b1y0²(12) where dig is the Kronecker delta, the 639 remainders in Eq. (13) vanish.
For example, the covariance of two neighboring feature map points in CNN with kernel size of is:
Cov(y1, U2) w w Cov(z), 222) + w W2 Cov (g) + w wa Cov(2,, 244)
+ W2 Cov(12, 22) + W2 Cov vta) + W2 Cov (22, T:4) wzy' Cov (xg, x22) W3W2 Cov (zg vaa) + W3 Cov (3:3, 2:4)
(14)
When v; tid, the covariance is:
Cov(y1, y2) = wh U2 0² (x2) + W2WWy o² (xx)
(15)
640
Since it is non-zero, the neighboring feature maps 91 and y2 are correlated.
Experiment. To demonstrate the spatial consistency of feature maps empirically, we provide feature map covariances randomly initialized single-layer CNN and five-layer CNN with ReLU lin-liiya In this experiment, the input values are Gaussian random noises. As shown in Fig. Da one convolutional layer correlates neighboring feature map points. Fig. D. shows that multiple convolutional layers correlate one feature map with distant feature maps. Moreover, the feature maps in deep CNNs have a stronger relationship with neighboring feature maps.
642
6M
646
D.2
ENSEMBLE FILTERS FREHUERRE SIGNALS
Following the notation of Eq. (3), the ensemble is convolution of importance л and prediction p:
P
(16)
24

---

Under review as a conference paper at ICLR 2022
where T2, xx,, and P. p(yl w.). To show that this ensemble is low-pass filter, apply the convolution times:
π * 7 xp
(17)
N times
648
π is the probability for the sum of N random variables from n, i.c, o + Φ ~ π *...... * π where & ~ л. By definition, an operator is low-pass filter if and only if the high frequency component vanishes when the operator is applied
Since π is probability, i... -
649
650
651
infinitely. Therefore, ensemble with π is low-pass filter because Var(ø + ..... + Φ) = N Var (o) and
652
F T
π pl = π * π F [)] where Fourier transform.
Experiment. Since blur filter is low-pas pass filter, probabilistic spatial smoothing is also low-pass filter. In Section 2.2, at the end of the stage 1, we show that MC dropout adds high-frequency noise to feature maps, and spatial smoothing effectively removes As shown in Fig. D.2, we observe the same phenomena at other stages.
654
655
656
657
In addition, Fig. 6C shows that CNNs are vulnerable high-freegg freeffreyeney random noise. Interestingly, also shows that CNNs are robust against noise with frequencies from 0.6T to 0.8T, corresponding to approximately 3 pixel periods. Since the receptive fields of convolutions are 3x3, the noise with a period smaller than the size is averaged out by convolutions. For the same reason, convolutions are particularly vulnerable against the noise with frequency of 0.3x, corresponding to a period pixel.
658
659
660
661
D.3 RANDOMNESS SHARPENS Loss LANDSCAPE, AND ENSEMBLE SMOOTHENS IT
664
Ws show that the randomness of BNNs hinder and destabilize NN training because it causes the loss landscape and its gradient to fluctuate from moment to moment. In other words, the randomness, such dropout, sharpens the loss landscape.
685
To show the claim theoretically, we use Foret et al. (2020)'s definition of sharpness with respect to training dataset DD:
sharpness, , = max LD + 5u)) (w(w) Sw
(18)
667 where Lo is NLL loss, w NN weight, dw is small weight perturbation, and is neighborhood radius. Therefore, dropout rate and the magnitude of increasss, the sharpness increases.
We next calculate the sharpness more rigorously. Let ps E (0, 1] be a confidence of one NN prediction, and p(N) be confidence of N ensemble, i.e., p(M) = N loss is:
1 Σ Pt. Then, the variance of the NLL
V = V
- log ((N)
(19)
D
D
V
log [T))
(20)
D
(1- DLN)
D V
log +
(21)
D(N)
W
(22)
[ 1 V [P.]
(23)
N D²(D)
(24)
N 1(
where μ = x and oeee is predictive variance of confidence. We use the formula v
Σm
N V [5] for arbitrary random variable &, and take the first-order Taylor expansion with an assump-
25

---

Under review as a conference paper at ICLR 2022
(a) Deterministic
(b) Deterministic Smooth
(c) MC dropout
(d) MC dropout + Smooth
6.0
Log ampplttdde
Log amplitude
Log amplitude
Log amplitadde
3.0
1.5
2.00
1.0
0.
0.0x
8.5x Frequency
100
8.0x
0.5x Frequency
1.0
0.x
0.5x
1.0n
0.0%
0.5x Frequency
1.0r
Frequency
Deterministie
Deterministic Smooth
MC dropout
MC dropout Smooth
(c) Diagonal components
Figure D.2: Spatial smoothing filters high-frequent signals including MC dropout noise. We present average feature maps ResNet-50 on ImageNet frequency space by using Fourier transform. Each column corresponds to feature maps stage to 4.
26

---

Under review as a conference paper at ICLR 2022
Log density
Density
A
20
200
400
600
Eigenvlaue
Max Eigenvlaue
0%
10%
20%
30 %
Figure D.3: Randomness due to MC dropout sharpens the loss function. We provide Hessian eigenvalue (left) and Hessian max eigenvalue spectra (right) of ResNet-18 on CIFAR-100.
tion p(N) 1 Eq. (21). Therefore, the approximated sharpness is:
pred
sharpness 2
(25)
N
1
669 In conclusion, the variance of NLL, (the square of) the sharpness, is proportional to the variance of predictions pred and inversely proportional to the ensemble size N. As the ensemble size increases 671 in the training phase, the loss landscape becomes smoother. Flat loss landscape results in better predictive performance and generalization (Foret et al., 2020).
Here, we only consider model uncertainty for the sake of simplicity. Extending the formulations to data uncertainty is straightforward. The predictive distribution of data-complemented BNN inference (Park et al., 2021) is:
p(y)S, / ply ar,
D)dddw
(26)
/
p(y plylz)p(z|S,2)) D)dx
(27)
673 where S is proximate data distribution, 2 (ax, w). and p(z D) p(x()(()((()))))))))) (D))) (D). This equation clearly shows that w and are symmetric. Therefore, obtain the formulas including both model and data uncertainty by replacing w with joint random variable of æ and w, i.e. w - z = (w, x).
675
Experiment.
Above, we claim two statements. First, the higher the dropout rate, the sharper the loss landscape. Second, the variance of the loss is inversely proportional to the ensemble size.
677
To demonstrate the former claim quantitatively, we compare the Hessian eigenvalue spectra and the Hessian max eigenvalue spectra of MC dropout with various dropout rates. In these experiments, ensemble size of one for MC dropout. For detailed explanation of Hessian max eigenvalue spectrum, see Appendix C.I.
679
682
Fig. D.3 represents the spectra, which reveals that as the randomness of the model increases, the number of Hessian eigenvalue outliers increases. Since outliers are detrimental to the optimization process (Ghorbani et al., 2019), dropout disturb NN optimization.
683
684
685
To show the latter claim, we evaluate the variance of NLL loss for ensemble size Nuain shown in Fig. D. As we would expect, the variance of the NLL loss-the sharpness of the loss landscape-is inversely proportional to the ensemble size for large N
686
D.4
TRAINING PHASE ENSEMBLE LEADS TO BETTER PERFORMANCE
689
Appendix D.3 raises an immediate question: Is there a performance difference between 'training with prediction ensemble' and "training with low MC dropout rate, instead of ensemble" ? Note
27

---

Under review as a conference paper at ICLR 2022
1.0c-07
NLLnnn variance
Nua =1
1.00
Nus =2
NLLtest
Nou
5.0c-08
0.90
10
10
10a
10
N rrain
N Nrest
(a) V [C] for ensemble size training dataset
(b) NLL for ensemble size test datasct
Figure D.4: Training phase ensemble helps NN learn strong representation. Left: The variance
of NLL (V [C] on training dataset is inversely proportional to the ensemble size for large N See Eq. (24). Right: Training phase ensemble improves the predictive performance test dataset.
that both methods reduce the sharpness of the loss landscape. This section answers the question by providing theoretical and experimental explanations that the ensemble in the training phase can 693 improve predictive performance.
691
According to Gal & Ghahramani (2016), the total predictive variance (in regression tasks) is:
pred
o model + sample
(28)
where of moxtl is model precision and - the lower bound of the predictive variance, i.e.:
Sample is NN prediction variance. Therefore, the model precision
2
model
(29)
The model precision depends only on the model architecture. For example, in the case MC dropout, of moda is proportional to the dropout rate (Gal & Ghahramani, 2016) as follows:
anodel X dropout rate
(30)
These suggest that model precision dominate predictive variance if the MC dropout rate is large enough, i.e. even if the number of ensembles is increased in the training phase, the predictive variance is almost the same. In contrast, decreasing the MC dropout rate reduces prediction diversity, and it obviously leads to performance degradation. Therefore, in the training phase, is better to ensemble predictions than lower the MC dropout rate. We believe that the training phase ensemble is strongly correlated with Batch Augmentation (Hoffer et al., 2020). We leave concrete analysis for future work.
695
697
689
700
The experiments below support the theoretical analysis. We train MC dropout by using training-phase ensemble method with various ensemble sizes Nirain-
Experiment.
701
702 As we would expect, Fig. D.4b shows that training phase ensemble significantly improves the predictive performance. In this experiment, we use MC dropout rate of 30%. As shown Fig. A.1,
703
provides the best predictive performance. We use ensemble size Nist 50 in test phase.
705
also measure the predictive variances of NLL. The predictive variances of the model with Nuan and with Nuin 3 are V [4] 0.0169 and V [4] = 0.0179 79, respectively. Since the predictive variances of the two models are almost the same, we infer that there exists lower bound.
707
708
E
EXTENDED INFORMATIONS OF EXPERIMENTS
709
This section provides additional information on the experiments in Section
710
E.
IMAGE CLASSIFICATION
711
We present numerical comparisons in the image classification experiment and discuss the results in detail.
28

---

Under review as a conference paper at ICLR 2022
Table E.I: Spatial smoothing improves both accuracy and uncertainty at the same time. Predic- tive performance of models with spatial smoothing in image classification CIFAR-IO, CIFAR-100, and ImageNet.
MODEL & DATASET
Acc: (%)
ECE (%)
MC DROPOUT
SMOOTH
NLL
0.401 (-0.000)
93.1 (+0.0)
3.80 (-0.00)
VGG-19 & CIFAR-10
0.376 (-0.002)
93.2 (+0.1)
5.49 (+1.69)
0.238 (-0.000)
92.6 (+0.0)
3.55 (-0.00)
0.197 (-0.041)
93.3 (+0.7)
0.68 (-2.86)
0.182 (-0.000)
95.2 (+0.0)
2.75 (-0.00)
ResNet-18 & CIFAR-10
0.173 (-0.009)
95.4 (+0.2)
2.31 (-0.44)
0.157 (-0.000)
95.2 (+0.0)
1.14 (-0.00)
0.144 (-0.014)
95.5 (+0.2)
1.04 (-0.10)
2.047 (-0.000)
71.6 (+0.0)
19.2 (-0.0)
VGG-16 1GG-16& & CIFAR-100
1.878 (-0.169)
72.2 (+0.6)
20.5 (+1.3)
1.133 (-0.000)
68.8 (+0.0)
3.66 (-0.00)
1.034 (-0.099)
71.4 (+2.6)
1.06 (-2.60)
2.016 (-0.000)
67.6 (+0.0)
21.2 (-0.0)
VGG-19
1.851 (-0.165)
71.7 (+4.0)
20.2 (-1.0)
CIFAR-100
1.215 (-0.000)
67.3 (+0.0)
6.37 (-0.00)
1.071 (-0.144)
70.4 (43.0)
2.15 (-4.22)
0.886 (-0.000)
77.9 (+0.0)
4.97 (-0.00)
ResNet-18 & CIFAR-100
0.863 (-0.023)
78.9 (+1.0)
4.40 (-0.57)
0.848 (-0.000)
77.3 (+0.0)
3.01 (-0.00)
0.801 (-0.047)
78.9 (+1.6)
2.56 (-0.45)
0.835 (-0.000)
79.9 (+0.0)
8.88 (-0.00)
ResNet-50 & CIFAR-100
0.834 (-0.002)
80.7 (+0.8)
9.29 (+0.42)
0.822 (-0.000)
79.1 (+0.0)
6.63 (-0.00)
0.800 (-0.022)
80.1 (+1.0)
7.25 (+0.62)
0.804 (-0.000)
80.6 (+0.0)
8.23 (-0.00)
ResNeXt-500 & CIFAR-100
0.825 (+0.022)
80.8 (+0.3)
9.41 (+1.18)
0.762 (-0.000)
80.5 (+0.0)
5.67 (-0.00)
0.759 (-0.002)
80.7 (+0.2)
6.62 (+0.94)
1.210 (-0.000)
70.3 (+0.0)
1.62 (-0.00)
ResNet-18 & ImageNet
1.183 (-0.027)
70.6 (+0.3)
1.22 (-0.40)
1.215 (-0.000)
70.0 (+0.0)
1.39 (-0.00)
1.190 (-0.032)
70.6 (+0.6)
2.25 (+0.86)
0.949 (-0.000)
76.0 (+0.0)
2.97 (-0.00)
ResNet-50 & ImageNet
0.916 (-0.033)
76.9 (+0.9)
3,46 (+0.49)
0.945 (-0.000)
76.0 (+0.0)
1.89 (-0.00)
0.905 (-0.040)
77.0 (+1.0)
2.49 (+0.60)
0.919 (-0.000)
77.7 (+0.0)
3.63 (-0.00)
ResNeXt-50 ImageNet
0.907 (-0.012)
78.0 (+0.3)
4.60 (+0.97)
0.895 (-0.000)
77.7 (+0.0)
2,53 (-0.00)
0.887 (-0.008)
78.1 (+0.4)
3.28 (+0.75)
29

---

Under review as a conference paper at ICLR 2022
0.40
Accuracy (%)
94.0
ECE (%)
4.0
NLL
0.30
92.0-
00
2.0
90.0
0.20
100
10"
10
10a
10
Ensemble Size
Ensemble Size
Ensemble Size
VGG-19 CIFAR-10
96.0
3.0
0.20
Accuracy
NLL
ECE
2.0
0.15
95.0-
1.0
10
10"
Ensemble Size
Ensemble Size
Ensemble Size
(b) ResNet- 18 CIFAR-10
82.0
8.0
1.00
Accuracy %%
80.0
ECF. (%)
6.0
NIJ
0.90
78.0
0.80
4.0
76.0
10
10'
10
10a
Ensemble Size
Ensemble Size
Ensemble Size
(c) ResNet-18 on CIFAR-100
1.00
10.0
82.0
ECE (96)
NLL
Accuracy
7.5-
0.80
80.0
5.0-
00
78.0
2.5
10
10
10a
10
10a
10"
Ensemble Size
Ensemble Size
Ensemble Size
(d) ResNet-50 on CIFAR-100
Accuracy
78.0
NLI
0.90
ECE
2.0
0.85
76.0
-0 -9-0
10"
10
10a
10a
10a
10a
Ensemble Size
Ensemble Size
Ensemble Size
(e) ResNet-50 on ImageNet
0
Deep ensemble
Deep ensemble Smooth
MC dropout
MC drapout Smooth
Figure E.I: Spatial smoothing improves both accuracy and uncertainty across whole range of
ensemble sizes.
30

---

Under review as a conference paper at ICLR 2022
6.0
75
Accuracy (%)
40-
ECE. (9)
NLL
2.0
Test
Test
2
Test
2
4
Intensity
Intensity
Intensity
Deterministic Smooth
dropout
dropout
mCNLL (%)
90.0
mCECE (%)
60-
MIC (%)
65
87.5
40-
60
10"
10a
10
10a
10a
10
Ensemble Size
Ensemble Size
Ensemble Size
Deterministic
Deterministic- Smooth
dropout
dropout Smooth
Figure E.3: Spatial smoothing improves corruption robustness. We measure the predictive per- formance of ResNet-18 CIFAR-100-C. In the top row, we use ensemble size of fifty for MC dropout with and without spatial smoothing.
713
Computational performance. The throughput of MC dropout and "MC dropout + spatial smooth- ing" is 755 and 675 image/sec, respectively, in training phase ImageNet. As mentioned in lines Section 3.1, NLL of "MC dropout spatial smoothing" with ensemble size of 2 is comparable even better than that of MC dropout with ensemble size of 50. Therefore, "MC dropout + spatial smoothing" is 22x faster than MC dropout with similar predictive performance, in terms of throughput.
715
717
Predictive performance on test dataset. Fig. E.2 repre- sents the reliability diagram of ResNet-18 18 on CIFAR-100 100, which shows that spatial smoothing improves the uncertainty of both deterministic and Bayesian NNs. Numerical compar- isons are provided below.
100
720
722
80
Accuracy
724
Table E. shows the predictive performance of various deter- ministic and Bayesian NNs with and without spatial smooth- ing on CIFAR-10, 10, CIFAR- 100, and ImageNet. This table suggests the following: First, spatial smoothing improves both accuracy and uncertainty in most cases. In particular, it improves the predictive performance of all models with MC dropouts. Second, spatial smoothing significantly improves the predictive performance of VGG compared with ResNet. VGG has chaotic loss landscape, which results in poor pre- dictive performance (Li al., 2018), and spatial smoothing smoothens its loss landscape effectively. Third, as the depth increases, the performance improvement decreases. Deeper NNs provide more overconfident results (Guo et al., 2017), but the number of spatial smoothing layers calibrating uncer- tainty is fixed. Last, the performance improvement of ResNeXt, which includes ensemble in its internal structure, is relatively marginal.
60
726
728
60
100
729
Confidence
730
Determinatic
MC deopout
Determinätic Smooth
732
733
Figure E.2: Spatial smoothing cal- ibrates predictions. We present re- liability diagram of ResNet-18 CIFAR-
735
737
739
Fig. E.1 shows predictive performance of MC dropout and deep ensemble for ensemble size. A deep ensemble with an ensemble size of a deterministic NN. This figure shows that spatial smoothing improves efficiency of ensemble size and the predictive performance at ensemble size of
741
31

---

Under review as a conference paper at ICLR 2022
Table E.2: Spatial smoothing improves adversarial robustness. We measure the accuracy (Acc)
and the Attack Success Rate (ASR) of ResNet-50 against adversarial attacks ImageNet.
AcC (%)
ASR (%)
ATTACK
MC DROPOUT
SMOOTH
28.3 (40.0)
62.9 (-0.0)
30.3 (+2.0)
60.5 (-2.4)
FGSM
30.3 (+0.0)
59.8 (-0.0)
32.6 (+2.3)
57.4 (-2.4)
7.5 (+0.0)
90.1 (-0.0)
9.0 (+1.4)
88.2
PGD
12.2 (40.0)
83.7 (-0.0)
13.7 (+1.5)
82.1 (-1.6)
743
50. addition, spatial smoothing stabilizes NN training. It reduces the variance of the performance, especially in VGG.
744
745
peculiarity of the results on ImageNet that spatial smoothing degrades ECE ResNet-50. is because spatial smoothing significantly improves the accuracy in this case, and there tends to be a trade-off between accuracy and ECE, e.g. as shown (Guo et al., 2017), Fig. A. and Fig. B.3. Instead, spatial smoothing shows the improvement in NLL,, another uncertainty metric.
747
749
Predictive performance on training datasets. Note that spatial smoothing helps NN learn strong representations. In other words, spatial smoothing does regularize NNs. For example, NLL ResNet-18 with MC dropout on CIFAR-100 training dataset 2.20 x 10 2 The NLL of the ResNet with spatial smoothing 1.94 10-2. In conclusion, spatial smoothing reduces the training loss.
751
753
Corruption robustness. We measure predictive performance on CIFAR-100-C (Hendrycks & Dietterich, 2019) in order evaluate the robustness of the models against intensities and 15 types of data corruption. The top row of Fig. E.3 shows the results as a box plot. The box plot shows the median, interquartile range (IQR). minimum, and maximum of predictive performance for types. They reveal that spatial smoothing improves predictive performance for corrupted data. In particular, spatial smoothing undoubtedly helps in predicting reliable uncertainty.
755
756
758
To summarize the performance of corrupted data in a single value, Hendrycks & Dietterich (2019) introduced corruption eиror (CE) for quantitative comparison. CE., which is CE for corruption type e and model f, as follows:
CF =
FJJ
AlexNet
(31)
where EY, is top-1 error of f for corruption type e and intensity i, and E.AleNNet is the error of AlexNet. Mean CE or mCE summarizes CE/ by averaging them over 15 corruption types such as Gaussian noise, brightness, and show. Likewise, to evaluate robustness in terms of uncertainty, we introduce corruption NLL (CNLL, 4) and corruption ECE (CECE, J) as follows:
5
CNLL
Σ NLL!/
Σ
NLL AlexNet 1.
(32)
and
CECE!
Σ ECD!
BCE
AlexNet
(33)
32

---

Under review as a conference paper at ICLR 2022
Table E.3: Spatial smoothing improves the consistency, robustness against shift-perturbation.
We measure the consistency of ResNet-18 on CIFAR-IO-P Deterministic NN with means deep ensemble.
CoNs (%)
CEC (x10-2)
MC DROPOUT
SMOOTH
N
97.9 (+0.0)
1.03 (-0.00)
98.2 (+0.3)
1.16 (+0.13)
5
98.7 (+0.0)
1.22 (-0.00)
5
98.9 (+0.2)
1.33 (+0.11)
50
98.2 (+0.0)
1.29 (-0.00)
50
98.4 (+0.2)
1.34 (+0.05)
where NLL, iee and ECE/ are NLL and BCE of for с and i, respectively, mCNLL and mCECE are averages over corruption types. Experimental results show that spatial smoothing improves the robustness against data corruption. See Fig. E.3 for the results.
759
760
761
782
The bottom row of Fig. E.3 shows mCNLL, mCE, and mCECE for ensemble size. They indicates that spatial smoothing improves not only the efficiency but corruption robustness across whole range of ensemble size.
783
764
785
Adversarial robustness. We show that spatial smoothing also improves adversarial robustness. First, measure the robustness, in terms of accuracy and attack success rate (ASR), of ResNet- 50 on ImageNet against popular adversarial attacks, namely FGSM (Goodfellow et al., 2015) and PGD (Madry al., 20.8). Table E.2 indicate that both MC dropout and spatial smoothing improve robustness against adversarial attacks.
767
789
Next, find out how spatial smoothing improves adversarial robustness. To this end, similar to Section 2.2, we measure the accuracy on the test datasets with frequency-basee adversarial FGSM attack. This experimental result shows that spatial
771
772
perturbations. In this experiment, smoothing is particularly robust against high frequency (2 0.3r) adversarial attacks. This is because spatial smoothing low-pass filter, we mentioned Section 2.2. Since the ResNet is vulnerable
773
774
775
against high frequency adversarial attack, an ef- fective defense of spatial smoothing against high frequency attacks significantly improves the robust- ness.
776
777
Relative confidence (%)
100
778
75
Consistency. To evaluate the translation invari- ance models, we use consistency (Hendrycks & Dietterich, 2019; Zhang, 2019a), metric represent- ing translation consistency for shift-translated data sequences {a1,.... , M+ J. follows:
25
-6
-4
-2
Consistency = M Σ 1(g(x;) = g(x++1))
(34)
Translation (px)
779 where g(x)) = arg max p a. D). Table E.3 pro- CIFAR-IO-F
Deterministic
MC dropout
vides consistency of R8sNet-18 (Hendrycks & Dietterich, 2019). The results shows that MC dropout and deep ensemble improve sistency, and spatial smoothing improves consis- tency of both deterministic and Bayesian NNs.
Determiniatic
Smooth
MC dropout Smooth
781
Figure E.4: Spatial smoothing improves the confidence when the predictions are incorrect.
783
We define relative confidence (See Eq. (36)), and measure the metric of ResNet- -8 on CIFAR- 10-P.
Prior works (Zhang, 2019a; Azulay & Weiss, 2019) investigated the fluctuation of predictive confidence
33

---

Under review as a conference paper at ICLR 2022
Table E.4: Spatial smoothing and temporal smoothing are complementary. We provide predictive performance of MC dropout in semantic segmentation CamVid for each method. SPAT and TEMP each stand for spatial smoothing and temporal smoothing. CONS stands for consistency.
ACC (%)
ECE (%)
CoNs (%)
MC DROPOUT
SPAT
TEMP
N
NLL
0.354 (+0.000)
92.3 (+0.0)
4.95 (+0.00)
95.1 (+0.0)
1
0.318 (+0.036)
92.4 (+0.1)
4.54 (40.41)
95.5 (40.4)
v
0.290 (+0.064)
92.5 (+0.2)
3.18 (+1.77)
96.3 (+1.2)
v
0.278 (+0.076)
92.5 (+0.2)
3.03 (+1.92)
96.6 (+1.5)
50
0.298 (+0.000)
92.5 (+0.0)
4.20 (40.00)
95.4 (+0.0)
V
50
0.284 (+0.014)
92.6 (+0.1)
3.96 (40.24)
95.6 (40.2)
V
1
0.273
(+0.025)
92.6 (+0.1)
3.23 (+0.97)
96.4 (+1.0)
v
0.260 (+0.038)
92.6 (+0.1)
2.71 (+1.49)
96.5 (+1.1)
on shift-translated data sequence. However, surprisingly find that confidence fluctuation has little to do with consistency. To demonstrate this claim, we introduce cross-entropy consistency (CEC, 4), metric that represents the fluctuation of confidence shift-translated data sequence {()1,--- 2M4+1 as follows:
1 CEC
M Σ f(x))) log(f
(35)
785
where f(x) p D). In Table E.3, high consistency does not mean low CEC; conversely, high consistency tends to be high CEC. Canonical NNs predict overconfident probabilities, and their confidence sometimes changes drastically from near-zero to near-one. Correspondingly, it results in
787
low consistency but low CEC. On the contrary, well-calibrated NNs such as MC dropout provide confidence that oscillates between zero and one, which results in high CEC.
789
To represent the NN reliability properly, we propose relative confidence (f) follows:
Relative confidence = p( x D)/ max p(yla xx, D)
(36)
where max p(y) x. D) is confidence of predictive result and p(yuue x; D) is probability of the result for true label. It is when NN classifies the image correctly, and less than when NN classifies incorrectly. Therefore, relative confidence is metric that indicates the overconfidence of prediction when NN's prediction is incorrect.
791
794
Figure E.4 shows qualitative example of consistency on CIFAR- 1IFAR-I0-PA by using relative confidence. This figure suggests that spatial smoothing improves consistency of both deterministic and Bayesian NN.
785
796
797
E.2
SEMANTIC SEGMENTATION
798
Table E.4 shows the performance of U-Net on the Cam Vid dataset. This table indicates that spatial smoothing improves accuracy, uncertainty, and consistency of deterministic and Bayesian NNs. This is consistent with the results in image classification. In addition, temporal smoothing leads to significant improvement in efficiency of ensemble size, accuracy, uncertainty, and consistency
800
802
by exploiting temporal information. Moreover, temporal smoothing requires only one ensemble to achieve high predictive performance, since it cooperates with the temporally previous predictions. We obtain the best predictive and computational performance by using both temporal smoothing and spatial smoothing.
804
34

---

Under review as a conference paper at ICLR 2022
F COMPARISON WITH ANTI-ALIASED CNN
807
As we mentioned in Scction 4, local mcans (Blur), also known as anti-aliased CNN (Zhang, 2019a), improve accuracy. Nevertheless, our work (Prob + Blur) has novelties in three respects: different motivation, improved uncertainty estimation, and analysis of how spatial smoothing works.
808
809
810
Different motivation. The motivation of local means was mitigate the aliasing effect of subsam- pling and to improve shift invariance. In contrast, our spatial smoothing is introduced to aggregate and ensemble nearby feature map points.
811
812
813
Improved uncertainty estimation. We demonstrate that spatial smoothing improves not only accuracy, but also uncertainty estimation and robustness against natural corruptions and adversarial attacks all at the same time. Moreover, we show that spatial smoothing significantly enhances the performance of MC dropout. Since there typically tends to be a trade-off between accuracy and shown in (Guo et al., 2017; Zhang et al., 2019; Geirhos et al.,
814
815
816
"uncertainty + robustness" e.g. 2019; Zhang, 2019b), Fig. I, and Fig. B.3-in NN modeling, we believe our simple yet effective method makes major inroads into the uncertainty quantification and generalization.
818
820
Analysis of how spatial smoothing improves performance. We find that the predictive perfor- mance improvement is not due to the anti-aliasing effect of local means.
822
Prob + Blur-our probabilistic spatial amoothing-impry actívation CNNs, but Blur alone-local mean or anti-aliased CNN-does not. In fact, contrary to (Zhang, 2019a), local mean degrades the predictive performance since it results in loss of information. It suggests that Prob plays an key role in prediction. For more details, see Appendix F.1.
the performance of pre-
823
824
825
826
827
Although the local filtering can result in loss of information, Zhang (2019a) experimentally observed increase in both thift-invariance (as expected) and accuracy (which was be- yond expectation). However, "there exist a fundamental trade-off between 'shift-invariance plus anti-aliasing' and performance" (Zhang, 2019b). Moreover, it is difficult relate anti-aliasing to improved uncertainty and robustness. Zhang (2019a) did not provide an explanation for these phenomena. As discussed in Appendix 1, spatial smoothing helps NNs learn strong representations, not regularizes NNs.
829
830
831
832
833
834
Spatial smoothing is, surprisingly, robust against blur corruptions.
We analyze how spatial smoothing improves predictive performance, by using loss landscape vi- sualization, Hessian eigenvalue spectra, and Fourier analysis. These analyzes draw the following conclusions:
836
838
Loss landscape visualization: Spatial smoothing stabilizes loss landscape fluc- tuations, caused by e.g. MC dropout. This results in stabilizing NN training and improving performance as well as generalization. See Figs. 8 and C.2. See also code/resources// /ooeees//sssslllaa httprepppeeeereeettteeeeeeeeeeee supplementary material.
839
840
811
1oloo_8..go0 gif and 18. in the
842
843
Hessian eigenvalue spectra: Spatial smoothing suppresses outliers of Hessian eigenvalues, which disrupt NN training. See Figs. 7 and C.3.
845
846
Fourier analysis: Spatial smoothing effectively removes high frequency signals, including noise due to MC dropout. We also show that CNNs are vulnerable to high frequency noise and high frequency adversarial attacks. See Figs. and D.2.
847
848
849
We also provide theoretical analysis of how spatial smoothing works. We prove that dropout sharpens the loss landscape, and ensemble smoothens it. Since the spatial smoothing is a spatial ensemble, significantly enhances the performance of MC dropout. See Appendix D.3 for more details. Further- more we also show that training -hase ensemble significantly improves the predictive performance because it smoothens the loss landscape without loss of prediction diversity. Therefore, the spatial smoothing, which ensembles feature map points at training time, improves the performance effectively. See Appendix D.4.
850
852
854
855
35

---

Under review as a conference paper at ICLR 2022
F.1
PROB PLAYS AN IMPORTANT ROLE SPATIAL SMOOTHING
857
As discussed in Section 2. we take the perspective that each point in feature map is prediction for binary classification by deriving the Bernoulli distributions from the feature map by using Prob. is in contrast to previous works known as sampling-fres BNNs (Hernández-Lobhe Wang et al.. 2016; Wu al., 2019) attempting to approximate the distribution of feature map with one Gaussian distribution. We do not use any assumptions on the distribution of feature map, and exactly represent the Bernoulli distributions and their averages. However, sampling-fres BNNs are emror prone because there is no guarantee that feature maps will follow a Gaussian distribution.
858
& Adams, 2015;
880
862
This Prob plays an important role in spatial smoothing. CNNs such as VGG, ResNet, and ResNeXt generally use post-activation arrangement. In other words, their stages end with BatchNorm and ReLU. Therefore, spatial smoothing layers Smooth(z) Blur Prob(z) in CNNs cooperates with BatchNorm and ReLU follows:
Prob(z) ReLU o tanh, o ReLU o BatchNorm (z)
(37)
ReLU tanh, BatchNorm (z)
(38)
since ReLU and tanh, commutative, and ReLU o ReLU is ReLU. This Prob is trainable and is general form of Eq. (7). we only use Blur as spatial smoothing, the activations BatchNorm ReLU
885
866
play the role of Prob.
867
In order to analyze the roles of Prob and Blur more precisely, measure the predictive perfor- mance of the model that does not use the post- activation. Figure F.l shows NLL of pre-activation 16 on CIFAR-100 The result shows that Blur with Prob improves the performance, but Blur alone does not. In fact, contrary to (Zhang, 2019a), blur degrades the predictive performance since it results in loss of information. We also measure the performance of VGG-19, ResNet- 18, ResNet-50, and BlurPool (Zhang, 2019a) with pre- activation, and observe the same phenomenon. In addition, BatchNorm- ReLU in front of GAP signif-
1.80
889
870
1.60
1.40
872
1.20
874
1.00
876
10"
10
Ensemble Size
878
Deterministic
dropout
dropout
880
icantly improves the performance of pre-activation ResNet.
Deterministic Prob
dropout
Figure F.l: Blur alone harms the predic- tive performance, although Prob + Blur im- proves it. We provide NLL of pre-activation VGG-16 on CIFAR-100.
882
As mentioned in Appendix C.2, pre-activation is special case of spatial smoothing. Therefore, the performance improvement of pre-activation by spa-
883
885
tial smoothing marginal compared that of post- activation.
36


Reviews:
[{"review_id": "m-wMSsDSNDg", "reviewer": "Reviewer_6pZ5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors have conducted many experiments to test their ideas, which is commendable. The paper is, however, too difficult to follow since much of these results are put in an appendix which significantly increases the page numbers, and for which, without the appendix, the paper does not have enough evidence. It is also difficult to follow the paper with the appendix referred to every 3rd sentence. The structure of the paper also lends to difficult reading, with results included in the methodology, figures out of place, the literature review at the end, and a too-brief conclusion.  ", "summary_of_the_paper": "The paper provides an improvement to Bayesian NNs computation needs and accuracy by incorporating spatial smoothing (blur). ", "summary_of_the_review": "I believe the content and ideas to be of interest but the paper requires a major revision and proper thought put into reducing the content without the appendix. \nThe language of the paper should also be improved. \nAcronyms are not defined throughout, assuming the reader knows what they stand for. \nThe references are mostly from arXiv - I do not find this appropriate. \nI am also not convinced the ideas are novel. The authors do not provide enough evidence to prove this. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bdXuyzGj0rJ", "reviewer": "Reviewer_3UhV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work explores the idea of spatial consistency to improve the computational cost of posteriors. The main idea is well presented and edge cases are explored. Although this is not my area of expertise, the theoretical analysis seems relevant and sound. Experimental results are convincing. Some more discussion on applications beyond the imaging domain could provide stronger significance to this method. The unifying perspective, when compared to global average pooling, pre-activation, and ReLU6, is a welcoming result.\n\n* Minor comments\n    - Figure 1 appears on page 1 without being referenced in the text", "summary_of_the_paper": "The motivation of this work is on the computational cost of using BNNs in practice, where applications might require a large number of BNNs in an ensemble formation for achieving good performance.\nThe work in this manuscript aims to reduce the computational cost ensemble.\n\nThe manuscript's insight for solving the computational cost is to exploit similarities in spatial neighbouring from images. This is a similar approach used in convolutional neural networks. Experimental results show improved efficiency in image tasks, while also considerably reducing the computational cost when compared to competitor methods.", "summary_of_the_review": "The work presents a novel method for reducing the computational cost when using BNNs in real-world image tasks.\nAlthough I am not an expert in the area, I judge the work as being worth publishing since the theoretical analysis seems relevant and experimental results convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "735g5SGxPSI", "reviewer": "Reviewer_RJkD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The spatial smooth layer is an interesting design that intuitively makes sense to be helpful for removing high-frequency feature noise in CNN especially with MC-dropout.\n- Authors also studied/explained their design from multiple angles, e.g. Fourier analysis of feature maps, changes to the loss landscape, understood as a spatial ensemble of BNN inference. \n- Authors did a great job of providing an extensive study on various tasks, datasets, and architectures.\n- This paper tries to build connections between their designs with some common practices in CNNs which are interesting and valuable for understanding why those designs are important for getting the performance.\n\nWeakness:\n- Smoothing the feature map for improving CNN's performance and robustness has been previously studied e.g. Sinha et a. NeurIPS2020 [1]. [1] even did the smoothing in a curriculum learning style. The authors should discuss related papers and compare them. The existing work also discounts the novelty of this paper.\n- This paper represents the spatial smooth design in the context of Bayesian neural network and claims that the smoothing e.g. averaging can be seen as an ensemble of neighboring features. I did not see strong support for this assumption. For computer vision task, different location of feature maps is usually be treated as independent variables. Using neighboring feature points for the ensemble is a coarse approximation even considering the neighboring pixels are often similar. Especially for downsampled feature maps, such an assumption can be dangerous. It is questionable if `smooth`'s improvement should be interpolated in this way. \n- I am concerned about why this technique is presented in a way strongly correlated with BNNs. After reading the introduction I got the impression that this paper's design is particularly for BNN although still helpful for deterministic models. However, only MC-dropout approximated BNN is studied in the paper. For other types of BNN, this technique seems not going to help. Considering there is no strong support for smoothing = ensembling, it may be more appropriate to understand the technique as removing the high-frequency noise inside the network and this issue is particularly important when dropout is present.\n- On the empirical value of the proposed technique. `Smooth` is more helpful are smaller datasets and architectures with irregular loss landscapes e.g. VGG but get marginal improvement or even hurts the ECE metric on ResNet (Table E.1). Smoothing feature maps can be dangerous for dense prediction tasks e.g. segmentation and detections. Not surprisingly, the limited results on semantic segmentation shown in the paper are weak. The limitations of the proposed method should be further discussed.\n\n[1] Sinha, Samarth, Animesh Garg, and Hugo Larochelle. \"Curriculum by smoothing.\" NeurIPS 2020", "summary_of_the_paper": "This paper proposed a spatial `smooth` layer including a feature range bounding layer `prob` and `blur` the intermediate feature map in a CNN. 'Smooth' improves the accuracy and uncertainty of both deterministic CNN and a Bayesian NN approximated by MC-dropout. Authors tried to justify how `smooth` improves the optimization of neural networks by 1. interpolating the `blur` operations as an ensemble of the neighboring features 2. showing `smooth` filter out the high-frequency noises introduced by MC-dropout and smoothen the loss landscapes perturbed by MC-dropout. Authors empirically evaluated `smooth` on image classification and semantic segmentation tasks and showed that it improves both accuracy and uncertainty. Authors also tried to connect common pieces in CNNs like global average pooling, ReLU + BN as special cases of `smooth`.", "summary_of_the_review": "This is a paper with simple designs but a lot of analysis and studies theoretically and empirically. The authors presented the proposed technique motivated in the context of Bayesian neural networks, which is not very strongly supported. The technique is clearly helpful in some settings and not very much in other settings. It needs further discussion on the limitation of the proposed technique and related works. I am slightly lean to reject this paper but may change my rating later.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0BlPPp5K-Uv", "reviewer": "Reviewer_vPUa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n \n+ The paper proposes spatial smoothing to alleviate the computational issues with Bayesian NNs.\n\n+ The paper empirically shows that spatial smoothing improves accuracy, uncertainty estimation, and robustness of BNNs across a whole range of ensemble sizes.\n\nWeaknesses:\n\n- The experimental evaluations are relatively weak or too some extent insufficient\n\n- The paper is not well self-contained, which highly relies on the analysis, experiments in the appendix.", "summary_of_the_paper": "The paper deals with the problem of uncertainty estimation. It proposed spatial smoothing, a method that ensembles neighboring feature map points of CNNs to alleviate the computational issues with Bayesian NNs. ", "summary_of_the_review": "While the theoretical analysis of the proposed spatial smoothing is interesting and valuable, the experimental evaluations are relatively weak or too some extent insufficient. Furthermore, the paper is not well self-contained, which highly relies on the analysis, experiments in the appendices.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a spatial smoothing layer for CNNs which is composed out of a feature range bounding layer (referred to as prob) and  a bluring layer (referred to as blur). An empirical analyses shows that the proposed layer improves the accuracy and uncertainty of both deterministic CNNs and Bayesian NN (BNNs) approximated by MC-dropout. The paper further provides theoretical arguments for the hypothesis that bluring corresponds to an ensemble and represents the proposed method as a strategy to reduce the sample amount during inference in BNNs.

Reviewers valued the extensive (theoretical as well as practical) analyses. However, the theoretical analysis should still be improved. First of all, the  the proposed technique is motivated in the context of BNNs, which is not very strongly supported. Second, the argument that „the smoothing layer is an ensemble“ is based on the observation that it has some properties ensembles have as well: (1) they reduce feature map variances, (2) filter out high-frequency signals, and (3) flatten the loss landscape. But two things sharing the same properties do not need to be the same thing. Moreover, the proofs of the prepositions stating the properties are difficult to follow and may contain some flaws. Furthermore, the paper is not well self-contained and highly depends on the appendix.
Given these, the paper can not be accepted in its current state. 

A future version could improve over the current manuscript by making the theoretical statements and proofs more clear. Another option would be to analyze the contribution without connecting it to a Bayesian setting and ensembles, and instead focus on showing that the proposed smoothing layer has those good properties, doing detailed empirical studies, and showing that CNN components like global average pooling and ReLU + BN are special cases of the propose method.
Decision: Reject