Year: 2021

Abstract:
Bayesian neural networks (BNNs) have shown success in the areas of uncertainty 1 estimation and robustness. However, a crucial challenge prohibits their use in 2 practice. Bayesian NNs require a large number of predictions to produce reliable 3 results, leading to a significant increase in computational cost. To alleviate this 4 issue, we propose spatial smoothing, a method that ensembles neighboring feature 5 map points of CNNs. By simply adding a few blur layers to the models, we 6 empirically show that spatial smoothing improves accuracy, uncertainty estimation, 7 and robustness of BNNs across a whole range of ensemble sizes. In particular, 8 BNNs incorporating spatial smoothing achieve high predictive performance merely 9 with a handful of ensembles. Moreover, this method also can be applied to canonical 10 deterministic neural networks to improve the performances. A number of evidences 11 suggest that the improvements can be attributed to the stabilized feature maps 12 and the flattening of the loss landscape. In addition, we provide a fundamental 13 explanation for prior works—namely, global average pooling, pre-activation, and 14 ReLU6—by addressing them as special cases of spatial smoothing. These not 15 only enhance accuracy, but also improve uncertainty estimation and robustness by 16 making the loss landscape smoother in the same manner as spatial smoothing. 17

text: Bayesian neural networks (BNNs) have shown success in the areas of uncertainty1 estimation and robustness. However, a crucial challenge prohibits their use in2 practice. Bayesian NNs require a large number of predictions to produce reliable3 results, leading to a significant increase in computational cost. To alleviate this4 issue, we propose spatial smoothing, a method that ensembles neighboring feature5 map points of CNNs. By simply adding a few blur layers to the models, we6 empirically show that spatial smoothing improves accuracy, uncertainty estimation,7 and robustness of BNNs across a whole range of ensemble sizes. In particular,8 BNNs incorporating spatial smoothing achieve high predictive performance merely9 with a handful of ensembles. Moreover, this method also can be applied to canonical10 deterministic neural networks to improve the performances. A number of evidences11 suggest that the improvements can be attributed to the stabilized feature maps12 and the flattening of the loss landscape. In addition, we provide a fundamental13 explanation for prior works—namely, global average pooling, pre-activation, and14 ReLU6—by addressing them as special cases of spatial smoothing. These not15 only enhance accuracy, but also improve uncertainty estimation and robustness by16 making the loss landscape smoother in the same manner as spatial smoothing.17
1 INTRODUCTION18
Azulay & Weiss, 2019).33
Bayesian neural networks (BNNs), such as Monte Carlo34 (MC) dropout (Gal & Ghahramani, 2016), provide a35 probabilistic representation of NN weights. They com-36 bine a number of models selected based on weight prob-37 ability to make predictions of desired results. Thanks to38
this feature, BNNs have been widely used in the areas of uncertainty estimation (Kendall & Gal, 2017)39 and robustness (Ovadia et al., 2019). They are also promising in other fields like out-of-distribution40 detection (Malinin & Gales, 2018) and meta-learning (Yoon et al., 2018).41
Nevertheless, there remains a significant challenge that prohibits their use in practice. BNNs require42 an ensemble size of up to fifty to achieve high predictive performance, which results in a fiftyfold43 increase in computational cost (Kendall & Gal, 2017; Loquercio et al., 2020). Therefore, if BNNs44 can achieve high predictive performance merely with a handful of ensembles, they could be applied45 to a much wider range of areas.46
1.1 PRELIMINARY47
We would first like to discuss BNN inference in detail, then move on to Vector-Quantized BNN48 (VQ-BNN) inference (Park et al., 2021), an efficient approximated BNN inference.49
BNN inference. Suppose we have access to posterior probability of NN weight p(w|D) for training dataset D. The predictive result of BNN is given by the following predictive distribution:
p(y|x0,D) = ∫ p(y|x0,w) p(w|D) dw (1)
wherex0 is observed input data vector, y is output vector, and p(y|x,w) is the probabilistic prediction parameterized by the result of NN for an input x and weight w. In most cases, the integral cannot be solved analytically. Thus, we use the MC estimator to approximate it as follows:
p(y|x0,D) ' N−1∑ i=0 1 N p(y|x0,wi) (2)
wherewi ∼ p(w|D) and N is the number of the samples. The equation indicates that BNN inference50 is ensemble average of NN predictions for one observed data point as shown on the left of Fig. 2.51 Using N neural networks in the ensemble would requires N times more computational complexity52 than one NN execution.53
1.00
Data-complemented BNN inference. To reduce the computational cost of BNN inference, VQBNN (Park et al., 2021) executes NN for an observed data only once and complements the result with previously calculated predictions for other data. If we have access to previous predictions, the computational performance of VQ-BNN becomes comparable to that of one NN execution. To be specific, VQ-BNN inference is:
p(y|x0,D) ' N−1∑ i=0 π(xi|x0) p(y|xi,wi) (3)
where π(xi|x0) is the importance of data xi with respect to the observed data x0, and it is defined as a54 similarity betweenxi andx0. p(y|x0,w0) is the newly calculated prediction, and {p(y|x1,w1), · · · }55 are previously calculated predictions. To accurately infer the results, the previous predictions should56 consist of predictions for “data similar to the observed data”.57
Thanks to the temporal consistency of real-world data streams, aggregating predictions for similar58 data in data streams is straightforward. Since temporally proximate data sequences tend to be similar,59 we can memorize recent predictions and calculates their average using exponentially decreasing60 importance. In other words, VQ-BNN inference for data streams is simply temporal smoothing of61 recent predictions as shown in the middle of Fig. 2.62
VQ-BNN has two limitations, although it may be a promising approach to obtain reliable results63 in an efficient way. First, it was only applicable to data streams such as video sequences. Applying64 VQ-BNN to images is challenging because it is impossible to memorize all similar images in advance.65 Second, Park et al. (2021) used VQ-BNN only in the testing phase, not in the training phase. We find66 that ensembling predictions for similar data helps in NN training by smoothing the loss landscape.67
1.2 MAIN CONTRIBUTION68
1 Spatially neighboring points in visual imagery tend to be similar, as do feature maps of convolu-69 tional neural networks (CNNs). By exploiting this spatial consistency, we propose spatial smoothing70 as a method of ensembling nearby feature maps to improve the efficiency of ensemble size in BNN71 inference. The right side of Fig. 2 visualizes spatial smoothing aggregating neighboring feature maps.72
2 We empirically demonstrate that spatial smoothing improves the efficiency in vision tasks, such73 as image classification on CIFAR (Krizhevsky et al., 2009) and ImageNet (Russakovsky et al., 2015),74 without any additional training parameters. Figure 3 shows that negative log-likelihood (NLL) of75 “MC dropout + spatial smoothing” with an ensemble size of two is comparable to that of vanilla MC76 dropout with an ensemble size of fifty. We also demonstrate that spatial smoothing improves accuracy,77 uncertainty, and robustness all at the same time. Figure 1 shows that spatial smoothing improves both78 the accuracy and uncertainty of various deterministic and Bayesian NNs with an ensemble size of79 fifty on CIFAR-100.80
3 Global average pooling (GAP) (Lin et al., 2014; Zhou et al., 2016), pre-activation (He et al.,81 2016b), and ReLU6 (Krizhevsky & Hinton, 2010; Sandler et al., 2018) have been widely used in vision82 tasks. However, their motives are largely justified by the experiments. We provide an explanation for83 these methods by addressing them as special cases of spatial smoothing. Experiments support the84 claim by showing that the methods improve not only accuracy but also uncertainty and robustness.85
2 PROBABILISTIC SPATIAL SMOOTHING86
To improve the computational perfor-87 mance of BNN inference, VQ-BNN88 (Park et al., 2021) executes NN pre-89 diction only once and complements90 the result with previously calculated91 predictions. The key to the success92 of this approach largely depends on93 the collection of previous predictions94 for proximate data. Gathering tempo-95 rally proximate data and their predic-96 tions from data streams is easy be-97 cause recent data and predictions can98 be aggregated using temporal consis-99 tency. On the other hand, gathering100
time-independent proximate data, e.g. images, is more difficult because they lack such consistency.101
2.1 MODULE ARCHITECTURE FOR ENSEMBLING NEIGHBORING FEATURE MAP POINTS102
So instead of temporal consistency, we use spatial consistency—where neighboring pixels of images103 are similar—for real-world images. Under this hypothesis, we take the feature maps as predictions104 and aggregate neighboring feature maps.105
Most CNN architectures, including ResNet, consist of multiple stages that begin with increasing the number of channels while reducing the spatial dimension of the input volume. We decompose an entire BNN inference into several steps by rewriting each stage in a recurrence relation as follows:
p(zi+1|zi,D) = ∫ p(zi+1|zi,wi) p(wi|D) dwi (4)
where zi is input volume of the i-th stage, and the first and the last volume are input data and106 output. wi and p(wi|D) are NN weight in the i-th stage and its probability. p(zi+1|zi,wi) is output107 probability of zi+1 with respect to the input volume zi. To derive the probability from the output108 feature map, we transform each point of the feature map into a Bernoulli distribution. To do so, a109 composition of tanh and ReLU, a function from value of range [−∞,∞] into probability, is added110 after each stage. Put shortly, we use neural networks for point-wise binary feature classification.111
Since Eq. (4) is a kind of BNN inference, it can be approximated using Eq. (3). In other words, each112 stage predicts feature map points only once and complements predictions with similar feature maps.113 Under spatial consistency, it averages probabilities of spatially neighboring feature map points, which114 is well known as blur operation in image processing. For the sake of implementation simplicity,115 average pooling with a kernel size of 2 and a stride of 1 is used as a box blur. This operation ensembles116 four neighboring probabilities with the same importances.117
In summary, as shown in Fig. 4, we propose the following probabilistic spatial smoothing layer:
Smooth(z) = Blur ◦ Prob (z) (5)
where Prob(·) is a point-wise function from a feature map to probability, and Blur(·) is importance-118 weighted average for ensembling spatially neighboring probabilities from feature maps. Smooth layer119 is added after each stage. Prob and Blur are further elaborated below.120
Prob: Feature map to probability. Prob is a function that transforms a real-valued feature map into probability. We use tanh–ReLU composition for this purpose. However, tanh is commonly known to suffer from the vanishing gradient problem. To alleviate this issue, we propose the following temperature-scaled tanh:
tanhτ (z) = τ tanh (z/τ) (6)
where τ is a hyperparameter called temperature. τ is 1 in conventional tanh and ∞ in identity121 function. tanhτ imposes an upper bound on a value, but does not limit the upper bound to 1.122

heading: MC dropout MC dropout + Smooth
text: An unnormalized probability, ranging from 0 to τ , is allowed as the output of Prob. Then, thanks to the linearity of integration, we obtain an unnormalized predictive distribution accordingly. Taking this into account, we propose the following Prob:
Prob(z) = ReLU ◦ tanhτ (z) (7)
where τ > 1. We empirically determine τ to minimize NLL, a metric that measures both accuracy123 and uncertainty. See Fig. B.3 for more detailed ablation studies. In addition, we expect upper-bounded124 functions, e.g., ReLU6(z) = ReLU ◦ min(z, 6) and feature map scaling z/τ with τ > 1 which125 is BatchNorm, to be able to replace tanhτ in Prob; and as expected, these alternatives improve126 uncertainty estimation in addition to accuracy. See Appendix C.2 and Appendix C.3 for detailed127 discussions on activation (ReLU ◦ BatchNorm) and ReLU6 as Prob.128
Blur: Averaging neighboring probabilities. Blur averages the probabilities from feature maps. We primarily use the average pool with a kernel size of 2 and a stride of 1 as the implementation of Blur for the sake of simplicity. Nevertheless, we could generalize Blur by using the following depth-wise convolution, which acts on each input channel separately, with non-trainable kernel
K = 1
||k||21 k ⊗ k> (8)
where k is a 1D matrix, e.g., k ∈ {(1) , (1, 1) , (1, 2, 1) , (1, 4, 6, 4, 1)}. Different ks derive different129 importances for neighboring feature maps. We empirically show that most Blurs improve the130 predictive performance and that optimalK varies by model. For more ablation studies, see Table B.2.131
2.2 HOW DOES SPATIAL SMOOTHING HELP OPTIMIZATION?132
We present theoretical and empirical aspects to show that spatial smoothing ensembles feature maps.133
Feature map variance. BNNs have two types of uncertainties: One is model uncertainty and the134 other is data uncertainty (Park et al., 2021). These randomnesses increase the variance of the feature135 maps. To demonstrate that spatial smoothing is an ensemble, we use the following proposition:136
Proposition 1. Ensembles reduce the variance of predictions.137 We omit the proof since it is straightforward. In our context, predictions are output feature maps of a138 stage. We investigate model and data uncertainties of the predictions along NN layers to show that139 spatial smoothing reduces the randomnesses and ensembles feature maps. Figure 5 shows the model140 uncertainty and data uncertainty of Bayesian ResNet including MC dropout layers. In this figure, the141
uncertainty of MC dropout’s feature map only accumulates, and almost monotonically increases in142 every NN layer. In contrast, the uncertainty of “MC dropout + spatial smoothing”’s feature map is143 significantly decreases at the end of stages, suggesting that the smoothing layers ensemble the feature144 map. In other words, they make the feature map more accurate and stabilized input volumes for the145 next stages. In addition, consistently, the spatial smoothing layer close to the last layer significantly146 improves performance because it reduces the uncertainty of predictions largely. See Fig. B.5 for more147 detailed results. Deterministic NNs do not have model uncertainty but data uncertainty. Therefore,148 spatial smoothing improves the performance of deterministic NNs as well as Bayesian NNs.149
Fourier analysis. We also analyze spatial smoothing through the lens of Fourier transform:150 Proposition 2. Ensembles filter high-frequency signals.151
The proof is provided in Eqs. (16) to (17). Figure 6b shows the 2D Fourier transformed output152 feature map at the end of the stage 1. This figure reveals that MC dropout almost does not affect153 low-frequency (< 0.3π) ranges, and it adds high-frequency (≥ 0.3π) noises. Since spatial smoothing154 is a low-pass filter, it effectively filters high-frequency signals, including the noises caused by MC155 dropout.156
We also find that CNNs are particularly vulnerable to high-frequency noises. To demonstrate this157 claim, following Shao et al. (2021), we measure accuracy with respect to data with frequency-based158 random noise xnoise = x0 +F−1 (F(δ) Mf ), where x0 is clean data, F(·) and F−1(·) are Fourier159 transform and inverse Fourier transform, δ is random noise, and Mf is frequency mask as shown160 in Fig. 6a. Figure 6c exhibits the results. In sum, high-frequency noises, including those caused by161 MC dropout, significantly impair accuracy. Spatial smoothing improves the robustness by effectively162 removing high-frequency noises.163
Loss landscape. Lastly, we show that the randomness hinders NN training as follows:164 Proposition 3. Randomness of predictions sharpens the loss landscape, and ensembles flatten it.165
The proof is provided in Eqs. (18) to (25). Since a sharp loss function disturbs NN optimization166 (Keskar et al., 2017; Santurkar et al., 2018; Foret et al., 2020), reducing the uncertainty helps NN167 learn strong representations. For example, training phase NN ensemble averages out the randomness,168 and it flattens the loss function. In consequence, an ensemble of BNN outputs in training phase169 significantly improves the predictive performance. See Fig. D.4 for numerical results. However, we170 do not use training phase ensemble because it significantly increases the training time. Instead, we171 use spatial smoothing as a method that ensembles feature maps without sacrificing training time.172
We visualizes the loss landscapes (Li et al., 2018), the contours of NLL on training dataset. Figure 8b173 shows that the loss landscapes of MC dropout fluctuate and have irregular surfaces due to the174
randomness. As Li et al. (2018); Foret et al. (2020) pointed out,175 this may lead to poor generalization and predictive performance.176 Spatial smoothing reduces randomness as discussed above, and177 spatial smoothing aids in optimization by stabilizing and flattening178 the loss landscape of BNN as shown in Fig. 8c.179
Furthermore, we use Hessian to quantitatively represent the sharp-180 ness of the loss landscapes. Figure 7 shows the Hessian max eigen-181 value spectra of the models in Fig. 8 with a batch size of 128, which182 reveals that spatial smoothing reduces the magnitude of Hessian183 eigenvalues and suppresses outliers. Since large Hessian eigenval-184 ues disturb NN training (Ghorbani et al., 2019), we come to the185 same conclusion that spatial smoothing helps NN optimization. See186 Appendix C.1 for a more detailed description of the configurations187 of the Hessian max eigenvalue spectra. In addition, from these188 observations, we propose the conjecture that the flatter the loss189 landscape, the better the uncertainty estimation, and vice versa.190
2.3 REVISITING GLOBAL AVERAGE POOLING191
The success of GAP classifier in image classification is192 indisputable. The initial motivation and the most widely193 accepted explanation for this success is that GAP prevents194 overfitting by using far fewer parameters than multi-layer195 perceptron (MLP) (Lin et al., 2014). However, we discover196 that the explanation is poorly supported. We compares197 GAP with other classifiers including MLP. Contrary to198 popular belief, Table 1 suggests that MLP does not overfit199 the training dataset. MLP underfits or gives comparable200 performance to GAP on the training dataset. On the test201
dataset, GAP provides better results compared with MLP. See Table C.1 for more detailed results.202
Our argument is that GAP is an extreme case of spatial smoothing. In other words, GAP is successful203 because it ensembles feature maps and smoothens the loss landscape to help optimization. To support204 this claim, we visualizes the loss landscape of MLP as shown in Fig. 8a. It is chaotic compared to205 that of GAP as shown in Fig. 8b. Hessian shows the consistent results as demonstrated by Fig. 7.206
3 EXPERIMENTS207
This section presents two experiments. The first experiment is image classification through which208 we show that spatial smoothing not only improves the ensemble efficiency, but also the accuracy,209 uncertainty, and robustness of both deterministic NN and MC dropout. The second experiment is210 semantic segmentation on data streams through which we show that spatial smoothing and temporal211 smoothing (Park et al., 2021) are complementary. See Appendix A for more detailed configurations.212
1.00
Three metrics are measured in these experiments: NLL (↓1), accuracy (↑), and expected calibration213 error (ECE, ↓) (Guo et al., 2017). NLL represents both accuracy and uncertainty, and is the most214 widely used as a proper scoring rule. ECE measures discrepancy between accuracy and confidence.215
3.1 IMAGE CLASSIFICATION216
This section mainly discuss ResNet (He et al., 2016a). Table E.1 also discuss other settings that217 show the same trend: e.g., VGG (Simonyan & Zisserman, 2015), ResNeXt (Xie et al., 2017),218 and pre-activation models (He et al., 2016a). Spatial smoothing also improves deep ensemble219 (Lakshminarayanan et al., 2017), another non-Bayesian probabilistic NN method. See Fig. E.1.220
Performance. Fig. 3 and Fig. 9 show the predictive performances of ResNet-18 on CIFAR-100221 and ResNet-50 on ImageNet, respectively. The results indicate that spatial smoothing improves both222 accuracy and uncertainty in many respects. Let us be more specific. First, spatial smoothing improves223 the efficiency of ensemble size. In these examples, the NLL of “MC dropout + spatial smoothing”224 with an ensemble size of 2 is comparable to or even better than that of MC dropout with an ensemble225 size of 50. In other words, “MC dropout + spatial smoothing” is 25× faster than MC dropout with226 a similar predictive performance. Second, the predictive performance of “MC dropout + spatial227 smoothing” is better than that of MC dropout, at an ensemble size of 50. Third, spatial smoothing228 improves the predictive performance of deterministic NN, as well as MC dropout.229
Robustness. To evaluate robustness against data corruption, we230 measure predictive performance of ResNet-18 on CIFAR-100-231 C (Hendrycks & Dietterich, 2019). This dataset consists of data232 corrupted by 15 different types, each with 5 levels of intensity233 each. We use mean corruption NLL (mCNLL, ↓), the averages234 of NLL over intensities and corruption types, to summarize the235 performance of corrupted data in a single value. See Eq. (32) for236 a more rigorous definition. Figure 10 shows that spatial smoothing237 not only improves the efficiency but also corruption robustness238 across a whole range of ensemble size. See Fig. E.3 for more239 details. Spatial smoothing also improves adversarial robustness240 and perturbation consistency (↑) (Hendrycks & Dietterich, 2019;241 Zhang, 2019a), shift-transformation invariance. See Table E.2,242 Table E.3, and Fig. E.4 for more details.243
3.2 SEMANTIC SEGMENTATION244
Table 2 summarizes the result of semantic segmentation on CamVid dataset (Brostow et al., 2008)245 that consists of real-world 360×480 pixels videos. The table shows that spatial smoothing improves246 predictive performance, which is consistent with the image classification experiment. Moreover, the247 result reveals that spatial smoothing and temporal smoothing (Park et al., 2021) are complementary.248 See Table E.4 for more results.249
1We use arrows to indicate which direction is better.
Spatial smoothing can be compared with prior works in the following areas.251
Anti-aliased CNNs. Local means (Zhang, 2019a; Zou et al., 2020; Vasconcelos et al., 2020; Sinha252 et al., 2020) were introduced for the shift-invariance of deterministic CNNs in image classification.253 They were motivated to prevent the aliasing effect of subsampling. Although the local filtering can254 result in a loss of information, Zhang (2019a) experimentally observed an increase in accuracy that255 was beyond expectation. We provide a fundamental explanation for this phenomenon: Local means256 are a spatial ensemble. An ensemble not only improves accuracy, but also uncertainty and robustness257 of deterministic and Bayesian NNs. In Fig. F.1, we also show that the predictive performance258 improvement is not due to anti-aliasing of local mean. See Appendix F for more discussion on local259 means. For a discussion on non-local means (Wang et al., 2018) and self-attention (Dosovitskiy et al.,260 2021), see Section 5.261
Sampling-free BNNs. Sampling-free BNNs (Hernández-Lobato & Adams, 2015; Wang et al.,262 2016; Wu et al., 2019) predict results based on a single or couple of NN executions. To this end, it is263 assumed that posterior and feature maps follow Gaussian distributions. However, the discrepancy264 between reality and assumption accumulates in every NN layer. Consequently, to the best of our265 knowledge, most of the sampling-free BNNs could only be applied to shallow models, such as LeNet,266 and were tested on small datasets. Postels et al. (2019) applied sampling-free BNNs to SegNet;267 nonetheless, Park et al. (2021) argued that they do not predict well-calibrated results.268
Efficient deep ensembles. Deep ensemble (Lakshminarayanan et al., 2017; Fort et al., 2019) is269 another probabilistic NN approach for predicting reliable results. BatchEnsemble (Wen et al., 2020;270 Dusenberry et al., 2020) ensembles over a low-rank subspace to make deep ensemble more efficient.271 Depth uncertainty network (Antoran et al., 2020) aggregates feature maps from different depths of272 a single NN to predict results efficiently. Despite being robust against data corruption, it provides273 weaker predictive performance compared to deterministic NN and MC dropout.274
5 DISCUSSION275
We propose spatial smoothing, a simple yet efficient module to improve BNN. Three different per-276 spectives, namely, feature map variance, Fourier analysis, and loss landscape, suggest that spatial277 smoothing ensembles feature maps. The limitation of spatial smoothing is that designing its compo-278 nents requires inductive bias. In other words, the optimal shape of the blur kernel is model-dependent.279 We believe this problem can be solved by introducing self-attention (Vaswani et al., 2017). Self-280 attentions for computer vision (Dosovitskiy et al., 2021; Touvron et al., 2021; Carion et al., 2020)281 can be deemed as trainable importance-weighted ensembles of feature maps. The observation that282 Transformers are more robust than expected (Bhojanapalli et al., 2021; Shao et al., 2021) supports this283 claim. Therefore, using self-attentions to generalize spatial smoothing would be a promising future284 work because it not only expands our work, but also helps deepen our understanding of self-attention.285
REPRODUCIBILITY STATEMENT286
To ensure reproducibility, we provide comprehensive resources, such as code and experimental details.287 The codebase will be released as open source under the Apache License 2.0. See the supplemental288 material for the code. Appendix A provides the specifications of all models used in this work. Detailed289 experimental setup including hyperparameters and ablation study are also available in Appendix A290 and Appendix B. De-facto image datasets are used for all experiments as described in Appendix A.291
REFERENCES292
Javier Antoran, James Allingham, and José Miguel Hernández-Lobato. Depth uncertainty in neural293 networks. Advances in Neural Information Processing Systems, 2020.294
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small295 image transformations? Journal of Machine Learning Research, 2019.296
Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, and297 Andreas Veit. Understanding robustness of transformers for image classification. In Proceedings298 of the IEEE/CVF International Conference on Computer Vision, 2021.299
Gabriel J Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and recog-300 nition using structure from motion point clouds. In European Conference on Computer Vision.301 Springer, 2008.302
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey303 Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer304 Vision. Springer, 2020.305
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas306 Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image307 is worth 16x16 words: Transformers for image recognition at scale. In International Conference308 on Learning Representations, 2021.309
Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snoek, Katherine Heller, Balaji310 Lakshminarayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1311 factors. In International Conference on Machine Learning. PMLR, 2020.312
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring313 the landscape of spatial robustness. In International Conference on Machine Learning. PMLR,314 2019.315
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization316 for efficiently improving generalization. In International Conference on Learning Representations,317 2020.318
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspec-319 tive. arXiv preprint arXiv:1912.02757, 2019.320
Jonathan Frankle, David J Schwab, and Ari S Morcos. Training batchnorm and only batchnorm:321 On the expressive power of random features in cnns. In International Conference on Learning322 Representations, 2021.323
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model324 uncertainty in deep learning. In International Conference on Machine Learning. PMLR, 2016.325
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and326 Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves327 accuracy and robustness. In International Conference on Learning Representations, 2019.328
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization329 via hessian eigenvalue density. In International Conference on Machine Learning. PMLR, 2019.330
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial331 examples. International Conference on Learning Representations, 2015.332
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola,333 Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet334 in 1 hour. arXiv preprint arXiv:1706.02677, 2017.335
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural336 networks. In International Conference on Machine Learning. PMLR, 2017.337
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image338 recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,339 2016a.340
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual341 networks. In European Conference on Computer Vision. Springer, 2016b.342
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common343 corruptions and perturbations. In International Conference on Learning Representations, 2019.344
José Miguel Hernández-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learning345 of bayesian neural networks. In International Conference on Machine Learning. PMLR, 2015.346
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment347 your batch: Improving generalization through instance repetition. In Proceedings of the IEEE/CVF348 Conference on Computer Vision and Pattern Recognition, 2020.349
A Kendall, V Badrinarayanan, and R Cipolla. Bayesian segnet: Model uncertainty in deep convolu-350 tional encoder-decoder architectures for scene understanding. In BMVC, 2017.351
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer352 vision? Advances in Neural Information Processing Systems, 2017.353
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter354 Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In355 International Conference on Learning Representations, 2017.356
Alex Krizhevsky and Geoff Hinton. Convolutional deep belief networks on cifar-10. Unpublished357 manuscript, 2010.358
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.359
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive360 uncertainty estimation using deep ensembles. In Advances in Neural Information Processing361 Systems, 2017.362
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape363 of neural nets. In Advances in Neural Information Processing Systems, 2018.364
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In International Conference on365 Learning Representations, 2014.366
Antonio Loquercio, Mattia Segu, and Davide Scaramuzza. A general framework for uncertainty367 estimation in deep learning. IEEE Robotics and Automation Letters, 2020.368
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.369 Towards deep learning models resistant to adversarial attacks. In International Conference on370 Learning Representations, 2018.371
A Malinin and M Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural372 Information Processing Systems. Curran Associates, Inc., 2018.373
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua374 Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model’s uncertainty?375 evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing376 Systems, 2019.377
Namuk Park, Taekyu Lee, and Songkuk Kim. Vector quantized bayesian neural network inference378 for data streams. In AAAI Conference on Artificial Intelligence, 2021.379
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor380 Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style,381 high-performance deep learning library. Advances in Neural Information Processing Systems,382 2019.383
Janis Postels, Francesco Ferroni, Huseyin Coskun, Nassir Navab, and Federico Tombari. Sampling-384 free epistemic uncertainty estimation using approximated variance propagation. In Proceedings of385 the IEEE/CVF International Conference on Computer Vision, 2019.386
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical387 image segmentation. In International Conference on Medical image computing and computer-388 assisted intervention. Springer, 2015.389
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang,390 Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet391 Large Scale Visual Recognition Challenge. International Journal of Computer Vision, pp. 211–252,392 2015.393
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo-394 bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on395 Computer Vision and Pattern Recognition, 2018.396
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal-397 ization help optimization? Advances in Neural Information Processing Systems, 2018.398
Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin-Yu Chen, and Cho-Jui Hsieh. On the adversarial robustness399 of visual transformers. arXiv preprint arXiv:2103.15670, 2021.400
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image401 recognition. In International Conference on Learning Representations, 2015.402
Samarth Sinha, Animesh Garg, and Hugo Larochelle. Curriculum by smoothing. Advances in Neural403 Information Processing Systems, 2020.404
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé405 Jégou. Training data-efficient image transformers & distillation through attention. In International406 Conference on Machine Learning. PMLR, 2021.407
Cristina Vasconcelos, Hugo Larochelle, Vincent Dumoulin, Nicolas Le Roux, and Ross Goroshin.408 An effective anti-aliasing approach for residual networks. arXiv preprint arXiv:2011.10675, 2020.409
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz410 Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information411 Processing Systems, 2017.412
Hao Wang, Xingjian Shi, and Dit-Yan Yeung. Natural-parameter networks: A class of probabilistic413 neural networks. Advances in Neural Information Processing Systems, 2016.414
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In415 Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.416
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient417 ensemble and lifelong learning. In International Conference on Learning Representations, 2020.418
Anqi Wu, Sebastian Nowozin, Edward Meeds, Richard E Turner, José Miguel Hernández-Lobato,419 and Alexander L Gaunt. Deterministic variational inference for robust bayesian neural networks.420 In International Conference on Learning Representations, 2019.421
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual422 transformations for deep neural networks. In Proceedings of the IEEE Conference on Computer423 Vision and Pattern Recognition, 2017.424
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks425 through the lens of the hessian. In 2020 IEEE International Conference on Big Data (Big Data).426 IEEE, 2020.427
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn.428 Bayesian model-agnostic meta-learning. In Advances in Neural Information Processing Systems,429 2018.430
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.431
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan.432 Theoretically principled trade-off between robustness and accuracy. In International Conference433 on Machine Learning. PMLR, 2019.434
Richard Zhang. Making convolutional networks shift-invariant again. In International Conference on435 Machine Learning. PMLR, 2019a.436
Richard Zhang. Official meta-review of making convolutional networks shift-invariant again, 2019b.437 URL https://openreview.net/forum?id=SklVEnR5K7&noteId=rklZnFS-gN.438
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep439 features for discriminative localization. In Proceedings of the IEEE Conference on Computer440 Vision and Pattern Recognition, 2016.441
Xueyan Zou, Fanyi Xiao, Zhiding Yu, and Yong Jae Lee. Delving deeper into anti-aliasing in convnets.442 In BMVC, 2020.443

heading: MC dropout (N=1) MC dropout (N=50) MC dropout + Smooth (N=1) MC dropout + Smooth (N=50)
text: A EXPERIMENTAL SETUP AND DATASETS444
We obtain the main experimental results with the Intel Xeon W-2123 Processor, 32GB memory, and445 a single GeForce RTX 2080 Ti for CIFAR (Krizhevsky et al., 2009) and CamVid (Brostow et al.,446 2008). For ImageNet (Russakovsky et al., 2015), we use AMD Ryzen Threadripper 3960X 24-Core447 Processor, 256GB memory, and four GeForce RTX 2080 Ti. We conduct ablation studies with four448 Intel Intel Broadwell CPUs, 15GB memory, and a single NVIDIA T4. Models are implemented449 in PyTorch(Paszke et al., 2019). The detailed configurations of image classification and semantic450 segmentation are as follows.451
A.1 IMAGE CLASSIFICATION452
We use VGG (Simonyan & Zisserman, 2015), ResNet (He et al., 2016a), pre-activation ResNet (He453 et al., 2016a), and ResNeXt (Xie et al., 2017) in image classification. According to the structure454 suggested by Zagoruyko & Komodakis (2016), each block of Bayesian NNs contains one MC dropout455 layer.456
NNs are trained using categorical cross-entropy loss and SGD optimizer with initial learning rate of457 0.1, momentum of 0.9, and weight decay of 5× 10−4. We also use multi-step learning rate scheduler458 with milestones at 60, 130, and 160, and gamma of 0.2 on CIFAR, and with milestones at 30, 60,459 and 80, and gamma of 0.2 on ImageNet. We train NNs for 200 epochs with batch size of 128 on460 CIFAR, and for 90 epochs with batch size of 256 on ImageNet. We start training with gradual warmup461 (Goyal et al., 2017) for 1 epoch on CIFAR. Basic data augmentations, namely random cropping and462 horizontal flipping, are used. One exception is the training of ResNeXt on ImageNet. In this case, we463 use the batch size of 128 and learning rate of 0.05 because of memory limitation.464
We use hyperparameters that minimizes NLL of ResNet. Table A.1 provides hyperparameters for465 deterministic and Bayesian NNs. For fair comparison, models with and without spatial smoothing466 share hyperparameters such as MC dropout rate. However, Fig. A.1 shows that spatial smoothing467 improves predictive performance of ResNet-18 at all dropout rates on CIFAR-100. The default468 ensemble size of MC dropout is 50. We report averages of three evaluations, and error bars in figures469 represent min and max values. Standard deviations are omitted from tables for better visualization.470 See the source code released on GitHub for other details.471
A.2 SEMANTIC SEGMENTATION472
We use U-Net (Ronneberger et al., 2015) in semantic segmentation. Following Bayesian SegNet473 (Kendall et al., 2017), Bayesian U-Net contains six MC dropout layers. We add spatial smoothing474 before each subsampling layer in U-Net encoder. We use 5 previous predictions and decay rate of475 e−0.8 for temporal smoothing.476
CamVid consists of 720×960 pixels road scene video sequences. We resize the image bilinearly to477 360×480 pixels. We use a list reduced to 11 labels by following previous works, e.g. (Kendall & Gal,478 2017).479
NNs are trained using categorical cross-entropy loss and Adam optimizer with initial learning rate of480 0.001 and β1 of 0.9, and β2 of 0.999. We train NN for 130 epoch with batch size of 3. The learning481 rate decreases to 0.0002 at the 100 epoch. Random cropping and horizontal flipping are used for482 data augmentation. Median frequency balancing is used to mitigate dataset imbalance. Other details483 follow Park et al. (2021).484
B ABLATION STUDY485
The probabilistic spatial smoothing proposed in this486 paper consists of two components: Prob and Blur. This487 section explores several candidates for each component488 and their properties.489
B.1 Prob: FEATURE MAPS TO PROBABILITIES490
We define Prob as a composition of an upper-491 bounded function and ReLU, a function that imposes492 the lower bound of zero. Fig. B.1 shows widely used493 upper-bounded functions: tanhτ (x) = τ tanh(x/τ),494 ReLU6(x) = min(max(x, 6), 0), and constant scaling495 which is x/τ .496
Table B.1 shows the predictive performance improve-497 ment by Prob with various upper-bounded functions on498
10

heading: MC dropout + Smooth (N=1) MC dropout + Smooth (N=50)
text: CIFAR-100. In this experiment, we use models with MC dropout, and τ = 5 for constant scaling. The499 results indicate that upper-bounded functions with ReLU tend to improve accuracy and uncertainty500 at the same time. In addition, they show that Prob and Blur are complementary. The best results501 are obtained when using both Prob and Blur. For the main experiments, we use the composition502 of tanhτ and ReLU as Prob, because the hyperparameter of constant scaling is highly dependent on503 dataset and model.504
Temperature. The characteristics of temperature-scaled tanh depends on τ . Figure B.2 plots505 tanhτ and their first derivatives with various temperatures. As shown in this figure, tanhτ has a506 couple of useful properties. First, tanhτ has an upper bound of τ . Second, the first derivative of507 tanhτ at x = 0 does not depend on τ .508
Fig. B.3 shows the predictive performance of ResNet-18 with MC dropout and spatial smoothing for509 the temperature on CIFAR-100. In this figure, the accuracy increases as the temperature increases. In510 terms of ECE, NN predicts more underconfident results as τ decreases. It is a misinterpretation that511 the result is overconfident at low τ because ECE is high. By definition, ECE relies on the absolute512 value of the difference between confidence and accuracy. In this example, at low τ , the accuracy is513 greater than the confidence, which leads to a high ECE. Moreover, at τ = 0.2, ECE with N = 50 is514 greater than that with N = 1, which means that the result is severely underconfident. NLL, a metric515 representing both accuracy and uncertainty, is minimized when the accuracy and the uncertainty are516 balanced. In conclusion, we set the default value of τ to 10.517
1.0

heading: None s1 s2 s3 s4
text: B.2 Blur: AVERAGING NEIGHBORING PROBABILITIES518
Blur is a depth-wise convolution with a kernel. The kernel given by Eq. (8) is derived from various519 ks such as k ∈ {(1) , (1, 1) , (1, 2, 1) , (1, 4, 6, 4, 1)}. In these examples, if |k| is 1, Blur is identity.520 If |k| is 2, Blur is a box blur, which is used in the main experiments. If |k| is 3 or 5, Blur is an521 approximated Gaussian blur.522
Table B.2 shows predictive performance of models using spatial smoothing with the kernels on523 CIFAR-100. This results show that most kernels improve both accuracy and uncertainty. However,524 the most effective kernel size depends on the model.525
B.3 POSITION OF SPATIAL SMOOTHING.526
As shown in Fig. 5, the magnitude of uncertainty tends to increase as the depth increases. Therefore,527 we expect that spatial smoothing close to the output layer will mainly drive performance improvement.528
We investigate the predictive performance of models with MC dropout using only one spatial smooth-529 ing layer. Figure B.5 shows the predictive performance of ResNet-18 with one spatial smoothing after530 each stage on CIFAR-100. The results suggest that spatial smoothing after s3 is the most important531 for improving performance. Surprisingly, spatial smoothing after s4 is the least important. This is532 because GAP, the most extreme case of spatial smoothing, already exists there.533
C REVISITING PRIOR WORKS534
As mentioned in Section 2, prior works—namely, GAP, pre-activation, and ReLU6—are spacial cases535 of spatial smoothing. This section discusses them in detail.536
C.1 GLOBAL AVERAGE POOLING537
The composition of GAP and a fully connected layer is the most popular classifier in classification538 tasks. The original motivation and the most widely accepted explanation for the success is that GAP539 classifier prevents overfitting because it uses significantly fewer parameters than MLP (Lin et al.,540 2014). To verify this claim, we measure the predictive performance of MLP, GAP, and global max541 pooling (GMaxP), a classifier that uses the same number of parameters as GAP, on training dataset.542
Predictive performance. Table C.1 shows the experimental results on the training and the test543 dataset of CIFAR-100, suggesting that the explanation is poorly supported. On both the training544 and the test dataset, most predictive performance of MLP is worse than that of GAP. It is a counter-545 intuitive result meaning that MLP do not overfit the training dataset. In addition, the performance546 improvement by GAP is remarkable in VGG, which has irregular loss landscape. The predictive547

heading: GAP MLP GMaxP GMedP

heading: GAP MLP GMaxP GMedP
text: performance of GMaxP is better than that of MLP, but worse than that of GAP. This shows that using548 fewer parameters partially helps to improve predictive performance; however, it is insufficient to549 explain the predictive performance improvement by GAP. Finally, global median pooling (GMedP)550 provides better predictive performance than GMaxP. It implies that using other noise reduction551 methods instead of average pooling helps to improve predictive performance.552
Robustness. To evaluate the robustness of the classifiers, we measure the predictive performance of553 ResNet-18 using MC dropout with the classifiers on CIFAR-100-C. Figure C.1 shows the experimental554 results. This figure suggests that MLP is not robust against data corruption, as we would expect. In555 terms of accuracy, the robustness of GMaxP and GMedP is relatively comparable to that of GAP;556 however, in terms of uncertainty, GAP is the most robust. These are consistent results with other557 spatial smoothing experiments.558
Loss landscape visualization. To understand the mechanism of GAP performance improvement,559 we investigate the loss landscape. Figure C.2 shows the loss landscape sequences of ResNet with560 MC dropout. In this figure, each sequence shares the bases, but they fluctuate due to the randomness561 of the MC dropout. Figure C.2a is the loss landscape of the model using MLP classifier instead of562 GAP classifier. The loss landscape is chaotic and irregular, resulting in hindering and destabilizing563 NN optimization. Fig. C.2b is loss landscape sequence of ResNet with GAP classifier. Since GAP564 ensembles all of the feature map points at the last stage, it flattens and stabilizes the loss landscape.565 Likewise, as shown in Fig. C.2c, spatial smoothing layers at the end of all stages also flattens and566 stabilizes the loss landscape.567
Hessian eigenvalue spectra. To evaluate the smoothness of the loss landscapes quantitatively,568 we also investigate their Hessians at the optimized weights. In particular, we calculate Hessian569 eigenvalue spectra (Ghorbani et al., 2019), distributions of Hessian eigenvalues, to show how spatial570 smoothing helps NN optimization. To this end, we try to use stochastic Lanczos quadrature algorithm571
implemented by Yao et al. (2020). However, the problem is that the model with MLP classifier572 requires a lot of memory while the algorithm is memory inefficient.573
In the training phase, we calculate the mean gradients with respect to mini-batches, rather than the574 entire dataset. Therefore, it may be reasonable to investigate the properties of the Hessian “mini-575 batch-wisely”. For that purpose, we propose a method, Hessian max eigenvalue spectra, that evaluates576 the distribution of “Hessian’s maximum eigenvalues for one mini-batch”. We use power iteration to577 produce only the greatest eigenvalue of the Hessian. This algorithm is easy to implement and requires578 significantly less memory and computational cost, compared with stochastic Lanczos quadrature579 with respect to entire dataset. With this method, we can investigate the Hessian of NNs with MLP580 classifiers, which would require a lot of GPU memory.581
Figure 7 shows the Hessian max eigenvalue spectra of GAP classi-582 fier models with and without spatial smoothing layers. As Li et al.583 (2018); Foret et al. (2020) and Appendix D.3 pointed out, Hessian584 eigenvalue outliers disturb NN training. This figure explicitly show585 that the GAP and spatial smoothing reduce the magnitude of the586 Hessian eigenvalues and suppress the outliers, which leads to the587 same result as the previous visualizations: GAP as well as spatial588 smoothing smoothen the loss landscape. In conclusion, averag-589 ing feature map points tends to help neural network optimization590 by smoothing, flattening, and stabilizing the loss landscape. We591 observe a similar phenomenon for deterministic NNs. We also592 evaluate the Hesse eigenvalue spectrum as shown in Fig. C.3, and593 it leads to the same conclusion.594
In these experiments, we use MLP incorporating dropout layers595 with a rate of 50% as the classifier. Since the dropout is one of596 the factors that makes MLP underfit the training dataset, we also597 evaluate MLP using dropouts with a rate of 0%. Nevertheless, the598 results still shows that the predictive performance of MLP is worse599
than that of GAP on the training dataset. Moreover, it severely degrades predictive performance of600 ResNet on the test dataset.601
C.2 PRE-ACTIVATION602
He et al. (2016b) experimentally showed that the pre-activation arrangement, in which the activation603 ReLU ◦ BatchNorm is placed before the convolution, improves the accuracy of ResNet. Since γs of604 most BatchNorms in CNNs are near-zero (Frankle et al., 2021), BatchNorms reduce the magnitude605 of feature maps. As shown in Fig. B.1, constant scaling is a non-trainable BatchNorm with no606 bias, and it also reduces the magnitude of feature map. In Table B.1, we show that constant scaling607 improves predictive performance. Considering the similarity between Prob with constant scaling and608 conventional activation, i.e., the similarity between ReLU◦ConstantScaling and ReLU◦BatchNorm,609 we find that the pre-activation arrangement improves uncertainty as well as accuracy, because610 convolutions act as a Blur.611
To show this, we change the post-activation of all layers to pre-activation, and measure the predictive612 performance. For ResNet, we follow the original paper by He et al. (2016b). Table C.2 shows613 the predictive performance of models with pre-activation. The results suggests that pre-activation614 improves both accuracy and uncertainty in most cases. For deterministic VGG-19, pre-activation615 significantly degrades accuracy but improves NLL. In conclusion, they imply that pre-activation is a616 special case of spatial smoothing.617
Santurkar et al. (2018) argued that BatchNorm helps in optimization by flattening the loss landscape.618 We show that spatial smoothing flattens and smoothens the loss landscape, which is a consistent619 explanation. It will be interesting to investigate if BatchNorm helps in ensembling feature maps.620
C.3 RELU6621
ReLU6 was experimentally introduced to improve predictive performance (Krizhevsky & Hinton,622 2010). Sandler et al. (2018) used “ReLU6 as the non linearity because of its robustness when used623
with low-precision computation”. In Table B.1, we show that ReLU6s at the end of stages helps to624 ensemble spatial information by transforming the feature map to Bernoulli distributions. Since spatial625 smoothing improves robustness against data corruption, it seems reasonable that ReLU6 is robust to626 low-precision computation. A more abundant investigation into this topic is promising future works.627
We measure the predictive performance of NNs using all activations as ReLU6 instead of ReLU.628 However, in contrast to the results in Table B.1, the results are not consistent. We speculate that the629 reason is that a lot of ReLU6s overly regularize NNs.630
D EXTENDED ANALYSIS OF HOW SPATIAL SMOOTHING WORKS631
This section provides further explanation of the analysis in Section 2.2.632
D.1 NEIGHBORING FEATURE MAPS IN CNNS ARE SIMILAR633
This work exploits the spatial consistency of feature maps, i.e., neighboring feature maps in CNNs634 are similar. Below, we theoretically and empirically prove the spatial consistency. Moreover, this635 spatial consistency of feature maps holds even if the input data is spatially inconsistent.636
Consider a single-layer convolutional neural network with one channel:
yi = [w ∗ x]i (9)
= k∑ l=1 wlxi−l+1 (10)
where ∗ is convolution operator with a kernel of size k, y is feature map output, w is kernel weight, and x is input random variable. Then, the covariance of two neighboring feature maps is:
Cov(yi, yi+1) = Cov( k∑ l=1 wlxi−l+1, k∑ m=1 wmxi−m+2) (11)
= k∑ l=1 k∑ m=1 wlwm Cov(xi−l+1, xi−m+2) (12)
= k−1∑ l=1 wlwl+1 σ 2(xi−l+2) + · · · (13)
where σ2(xi−l+1) is the variance of xi−l+1. Therefore, Cov(yi,yi+1) is non-zero for randomly637 initialized weights. If x is iid, i.e., Cov(xi, xj) = δijσ2(xi) where δij is the Kronecker delta, the638 remainders in Eq. (13) vanish.639
For example, the covariance of two neighboring feature map points in a CNN with a kernel size of 3 is:
Cov(y1, y2) = w1w1 Cov(x1, x2) + w1w2 Cov(x1, x3) + w1w3 Cov(x1, x4) + w2w1 Cov(x2, x2) + w2w2 Cov(x2, x3) + w2w3 Cov(x2, x4) + w3w1 Cov(x3, x2) + w3w2 Cov(x3, x3) + w3w3 Cov(x3, x4)
(14)
When xi is iid, the covariance is:
Cov(y1, y2) = w1w2 σ2(x2) + w2w3 σ2(x3) (15)
Since it is non-zero, the neighboring feature maps y1 and y2 are correlated.640
Experiment. To demonstrate the spatial consistency of feature maps empirically, we provide641 feature map covariances of randomly initialized single-layer CNN and five-layer CNN with ReLU642 non-linearity. In this experiment, the input values are Gaussian random noises. As shown in Fig. D.1a,643 one convolutional layer correlates neighboring feature map points. Fig. D.1b shows that multiple644 convolutional layers correlate one feature map with distant feature maps. Moreover, the feature maps645 in deep CNNs have a stronger relationship with neighboring feature maps.646
D.2 ENSEMBLE FILTERS HIGH-FREQUENCY SIGNALS647
Following the notation of Eq. (3), the ensemble is convolution of importance π and prediction p:
π ∗ p (16)
where πi,j = π(xi|xj) and pi = p(y|xi,wi). To show that this ensemble is low-pass filter, we apply the convolution N times:
π ∗ · · · ∗ π︸ ︷︷ ︸ N times ∗p (17)
Since π is probability, i.e., ∑ i πi,j = 1, π ∗ · · · ∗ π is the probability for the sum of N random648 variables from π, i.e, φ + · · · + φ ∼ π ∗ · · · ∗ π where φ ∼ π. By definition, an operator is649 low-pass filter if and only if the high frequency component vanishes when the operator is applied650 infinitely. Therefore, ensemble with π is low-pass filter because Var(φ+ · · ·+ φ) = N Var(φ) and651 F [π ∗ · · · ∗ π ∗ p] = F [π ∗ · · · ∗ π] F [p] where F is Fourier transform.652
Experiment. Since blur filter is low-pass filter, probabilistic spatial smoothing is also low-pass653 filter. In Section 2.2, at the end of the stage 1, we show that MC dropout adds high-frequency noise to654 feature maps, and spatial smoothing effectively removes it. As shown in Fig. D.2, we observe the655 same phenomena at other stages.656
In addition, Fig. 6c shows that CNNs are vulnerable to high-frequency random noise. Interestingly, it657 also shows that CNNs are robust against noise with frequencies from 0.6π to 0.8π, corresponding to658 approximately 3 pixel periods. Since the receptive fields of convolutions are 3×3, the noise with a659 period smaller than the size is averaged out by convolutions. For the same reason, convolutions are660 particularly vulnerable against the noise with a frequency of 0.3π, corresponding to a period of 6661 pixel.662
D.3 RANDOMNESS SHARPENS LOSS LANDSCAPE, AND ENSEMBLE SMOOTHENS IT663
Ws show that the randomness of BNNs hinder and destabilize NN training because it causes the loss664 landscape and its gradient to fluctuate from moment to moment. In other words, the randomness,665 such as dropout, sharpens the loss landscape.666
To show the claim theoretically, we use Foret et al. (2020)’s definition of sharpness with respect to training dataset D:
sharpnessρ = max δw≤ρ LD(w + δw)− LD(w) (18)
where LD is NLL loss, w is NN weight, δw is small weight perturbation, and ρ is neighborhood667 radius. Therefore, as dropout rate—and the magnitude of δw—increases, the sharpness increases.668
We next calculate the sharpness more rigorously. Let pi ∈ (0, 1] be a confidence of one NN prediction, and p̄(N) be a confidence of N ensemble, i.e., p̄(N) = 1N ∑N i=1 pi. Then, the variance of the NLL loss is:
V [L] = V
[ 1
|D| ∑ D − log p̄(N)
] (19)
= 1 |D| V [ − log p̄(N) ] (20)
' 1 |D|
V [ − logµ+ ( 1− p̄ (N)
µ
)] (21)
= 1 |D| V [ − p̄ (N) µ ] (22)
= 1
N V [pi] µ2|D|
(23)
= 1
N σ2pred µ2|D|
(24)
where µ = p̄(∞) and σ2pred is predictive variance of confidence. We use the formula V [ 1 N ∑N i=1 ξ ] =
1 NV [ξ] for arbitrary random variable ξ, and we take the first-order Taylor expansion with an assump-
tion p̄(N) ' µ in Eq. (21). Therefore, the approximated sharpness is:
sharpness2ρ ' 1
N σ2pred µ2|D|
(25)
In conclusion, the variance of NLL, (the square of) the sharpness, is proportional to the variance of669 predictions σ2pred and inversely proportional to the ensemble size N . As the ensemble size increases670 in the training phase, the loss landscape becomes smoother. Flat loss landscape results in better671 predictive performance and generalization (Foret et al., 2020).672
Here, we only consider model uncertainty for the sake of simplicity. Extending the formulations to data uncertainty is straightforward. The predictive distribution of data-complemented BNN inference (Park et al., 2021) is:
p(y|S,D) = ∫ p(y|x,w)p(x|S)p(w|D)dxdw (26)
= ∫ p(y|z)p(z|S,D)dz (27)
where S is proximate data distribution, z = (x,w), and p(z|S,D) = p(x|S) p(w|D). This equation673 clearly shows that w and x are symmetric. Therefore, we obtain the formulas including both model674 and data uncertainty by replacing w with joint random variable of x and w, i.e. w → z = (w,x).675
Experiment. Above, we claim two statements. First, the higher the dropout rate, the sharper the676 loss landscape. Second, the variance of the loss is inversely proportional to the ensemble size.677
To demonstrate the former claim quantitatively, we compare the Hessian eigenvalue spectra and the678 Hessian max eigenvalue spectra of MC dropout with various dropout rates. In these experiments,679 we use ensemble size of one for MC dropout. For detailed explanation of Hessian max eigenvalue680 spectrum, see Appendix C.1.681
Fig. D.3 represents the spectra, which reveals that as the randomness of the model increases, the682 number of Hessian eigenvalue outliers increases. Since outliers are detrimental to the optimization683 process (Ghorbani et al., 2019), dropout disturb NN optimization.684
To show the latter claim, we evaluate the variance of NLL loss for ensemble size Ntrain as shown in685 Fig. D.4a. As we would expect, the variance of the NLL loss—the sharpness of the loss landscape—is686 inversely proportional to the ensemble size for large Ntrain.687
D.4 TRAINING PHASE ENSEMBLE LEADS TO BETTER PERFORMANCE688
Appendix D.3 raises an immediate question: Is there a performance difference between ‘training689 with prediction ensemble’ and ‘training with a low MC dropout rate, instead of no ensemble’? Note690
that both methods reduce the sharpness of the loss landscape. This section answers the question691 by providing theoretical and experimental explanations that the ensemble in the training phase can692 improve predictive performance.693
According to Gal & Ghahramani (2016), the total predictive variance (in regression tasks) is:
σ2pred = σ 2 model + σ 2 sample (28)
where σ2model is model precision and σ 2 sample is NN prediction variance. Therefore, the model precision is the lower bound of the predictive variance, i.e.:
σ2pred ≥ σ2model (29)
The model precision depends only on the model architecture. For example, in the case of MC dropout, σ2model is proportional to the dropout rate (Gal & Ghahramani, 2016) as follows:
σ2model ∝ dropout rate (30)
These suggest that model precision dominate predictive variance if the MC dropout rate is large694 enough, i.e., even if the number of ensembles is increased in the training phase, the predictive variance695 is almost the same. In contrast, decreasing the MC dropout rate reduces prediction diversity, and it696 obviously leads to performance degradation. Therefore, in the training phase, it is better to ensemble697 predictions than to lower the MC dropout rate. We believe that the training phase ensemble is strongly698 correlated with Batch Augmentation (Hoffer et al., 2020). We leave concrete analysis for future work.699
Experiment. The experiments below support the theoretical analysis. We train MC dropout by700 using training-phase ensemble method with various ensemble sizes Ntrain.701
As we would expect, Fig. D.4b shows that training phase ensemble significantly improves the702 predictive performance. In this experiment, we use MC dropout rate of 30%. As shown in Fig. A.1, it703 provides the best predictive performance. We use ensemble size Ntest = 50 in test phase.704
We also measure the predictive variances of NLL. The predictive variances of the model with705 Ntrain = 1 and with Ntrain = 3 are V [L] = 0.0169 and V [L] = 0.0179, respectively. Since the706 predictive variances of the two models are almost the same, we infer that there exists a lower bound.707
E EXTENDED INFORMATIONS OF EXPERIMENTS708
This section provides additional information on the experiments in Section 3.709
E.1 IMAGE CLASSIFICATION710
We present numerical comparisons in the image classification experiment and discuss the results in711 detail.712
Table E.1: Spatial smoothing improves both accuracy and uncertainty at the same time. Predictive performance of models with spatial smoothing in image classification on CIFAR-10, CIFAR-100, and ImageNet.
MODEL & DATASET MC DROPOUT SMOOTH NLL
ACC (%)
ECE (%)
VGG-19 & CIFAR-10
· · 0.401 (-0.000) 93.1 (+0.0) 3.80 (-0.00) · X 0.376 (-0.002) 93.2 (+0.1) 5.49 (+1.69) X · 0.238 (-0.000) 92.6 (+0.0) 3.55 (-0.00) X X 0.197 (-0.041) 93.3 (+0.7) 0.68 (-2.86)
ResNet-18 & CIFAR-10
· · 0.182 (-0.000) 95.2 (+0.0) 2.75 (-0.00) · X 0.173 (-0.009) 95.4 (+0.2) 2.31 (-0.44) X · 0.157 (-0.000) 95.2 (+0.0) 1.14 (-0.00) X X 0.144 (-0.014) 95.5 (+0.2) 1.04 (-0.10)
VGG-16 & CIFAR-100
· · 2.047 (-0.000) 71.6 (+0.0) 19.2 (-0.0) · X 1.878 (-0.169) 72.2 (+0.6) 20.5 (+1.3) X · 1.133 (-0.000) 68.8 (+0.0) 3.66 (-0.00) X X 1.034 (-0.099) 71.4 (+2.6) 1.06 (-2.60)
VGG-19 & CIFAR-100
· · 2.016 (-0.000) 67.6 (+0.0) 21.2 (-0.0) · X 1.851 (-0.165) 71.7 (+4.0) 20.2 (-1.0) X · 1.215 (-0.000) 67.3 (+0.0) 6.37 (-0.00) X X 1.071 (-0.144) 70.4 (+3.0) 2.15 (-4.22)
ResNet-18 & CIFAR-100
· · 0.886 (-0.000) 77.9 (+0.0) 4.97 (-0.00) · X 0.863 (-0.023) 78.9 (+1.0) 4.40 (-0.57) X · 0.848 (-0.000) 77.3 (+0.0) 3.01 (-0.00) X X 0.801 (-0.047) 78.9 (+1.6) 2.56 (-0.45)
ResNet-50 & CIFAR-100
· · 0.835 (-0.000) 79.9 (+0.0) 8.88 (-0.00) · X 0.834 (-0.002) 80.7 (+0.8) 9.29 (+0.42) X · 0.822 (-0.000) 79.1 (+0.0) 6.63 (-0.00) X X 0.800 (-0.022) 80.1 (+1.0) 7.25 (+0.62)
ResNeXt-50 & CIFAR-100
· · 0.804 (-0.000) 80.6 (+0.0) 8.23 (-0.00) · X 0.825 (+0.022) 80.8 (+0.3) 9.41 (+1.18) X · 0.762 (-0.000) 80.5 (+0.0) 5.67 (-0.00) X X 0.759 (-0.002) 80.7 (+0.2) 6.62 (+0.94)
ResNet-18 & ImageNet
· · 1.210 (-0.000) 70.3 (+0.0) 1.62 (-0.00) · X 1.183 (-0.027) 70.6 (+0.3) 1.22 (-0.40) X · 1.215 (-0.000) 70.0 (+0.0) 1.39 (-0.00) X X 1.190 (-0.032) 70.6 (+0.6) 2.25 (+0.86)
ResNet-50 & ImageNet
· · 0.949 (-0.000) 76.0 (+0.0) 2.97 (-0.00) · X 0.916 (-0.033) 76.9 (+0.9) 3.46 (+0.49) X · 0.945 (-0.000) 76.0 (+0.0) 1.89 (-0.00) X X 0.905 (-0.040) 77.0 (+1.0) 2.49 (+0.60)
ResNeXt-50 & ImageNet
· · 0.919 (-0.000) 77.7 (+0.0) 3.63 (-0.00) · X 0.907 (-0.012) 78.0 (+0.3) 4.60 (+0.97) X · 0.895 (-0.000) 77.7 (+0.0) 2.53 (-0.00) X X 0.887 (-0.008) 78.1 (+0.4) 3.28 (+0.75)
100 101 Ensemble Size
0.20
0.30 0.40 N LL
100 101 Ensemble Size
90.0
92.0
94.0
A cc
ur ac
y (%
)
100 101 Ensemble Size
2.0
4.0
6.0
EC E
(% )
(a) VGG-19 on CIFAR-10
100 101 Ensemble Size
0.15
0.20
N LL
100 101 Ensemble Size
95.0
96.0
A cc
ur ac
y (%
)
100 101 Ensemble Size
1.0
2.0
3.0
EC E
(% )
(b) ResNet-18 on CIFAR-10
100 101 Ensemble Size
0.80
0.90
1.00
N LL
100 101 Ensemble Size
76.0
78.0
80.0
82.0
A cc
ur ac
y (%
)
100 101 Ensemble Size
4.0
6.0
8.0
EC E
(% )
(c) ResNet-18 on CIFAR-100
100 101 Ensemble Size
0.80
1.00
N LL
100 101 Ensemble Size
78.0
80.0
82.0
A cc
ur ac
y (%
)
100 101 Ensemble Size
2.5
5.0
7.5
10.0
EC E
(% )
(d) ResNet-50 on CIFAR-100
100 101 Ensemble Size
0.80
0.85
0.90
0.95
N LL
100 101 Ensemble Size
76.0
78.0
A cc
ur ac
y (%
)
100 101 Ensemble Size
1.0
2.0
3.0
EC E
(% )
(e) ResNet-50 on ImageNet
0.00 0.25 0.50 0.75 1.00
0.00
0.25
0.50
0.75
1.00
Deep ensemble Deep ensemble + Smooth MC dropout MC dropout + Smooth
Figure E.1: Spatial smoothing improves both accuracy and uncertainty across a whole range of ensemble sizes.
Test 1 2 3 4 5 Intensity
2.0
4.0
6.0
N LL
Test 1 2 3 4 5 Intensity
25
50
75
A cc
ur ac
y (%
)
Test 1 2 3 4 5 Intensity
0
20
40
EC E
(% )
0 1 2 3 4 5 Intensity
0
10
20
EC E
(% )
Deterministic Deterministic + Smooth MC dropout MC dropout + Smooth
100 101 Ensemble Size
60
65
70
m CN
LL (%
)
100 101 Ensemble Size
85.0
87.5
90.0
m CE
(% )
100 101 Ensemble Size
40
60
m CE
CE (%
)
0.00 0.25 0.50 0.75 1.00
0.00
0.25
0.50
0.75
1.00
Deterministic Deterministic + Smooth MC dropout MC dropout + Smooth
Figure E.3: Spatial smoothing improves corruption robustness. We measure the predictive performance of ResNet-18 on CIFAR-100-C. In the top row, we use an ensemble size of fifty for MC dropout with and without spatial smoothing.
Computational performance. The throughput of MC dropout and “MC dropout + spatial smooth-713 ing” is 755 and 675 image/sec, respectively, in training phase on ImageNet. As mentioned in lines714 Section 3.1, NLL of “MC dropout + spatial smoothing” with ensemble size of 2 is comparable715 to or even better than that of MC dropout with ensemble size of 50. Therefore, “MC dropout +716 spatial smoothing” is 22× faster than MC dropout with similar predictive performance, in terms of717 throughput.718
40 60 80 100 Confidence (%)
40
60
80
100
A cc
ur ac
y (%
)
0.00 0.25 0.50 0.75 1.00
0.00
0.25
0.50
0.75
1.00
Deterministic Deterministic + Smooth
MC dropout MC dropout + Smooth
Figure E.2: Spatial smoothing calibrates predictions. We present reliability diagram of ResNet-18 on CIFAR-100.
Predictive performance on test dataset. Fig. E.2 repre-719 sents the reliability diagram of ResNet-18 on CIFAR-100,720 which shows that spatial smoothing improves the uncertainty721 of both deterministic and Bayesian NNs. Numerical compar-722 isons are provided below.723
Table E.1 shows the predictive performance of various deter-724 ministic and Bayesian NNs with and without spatial smooth-725 ing on CIFAR-10, CIFAR-100, and ImageNet. This table726 suggests the following: First, spatial smoothing improves727 both accuracy and uncertainty in most cases. In particular, it728 improves the predictive performance of all models with MC729 dropouts. Second, spatial smoothing significantly improves730 the predictive performance of VGG compared with ResNet.731 VGG has a chaotic loss landscape, which results in poor pre-732 dictive performance (Li et al., 2018), and spatial smoothing733 smoothens its loss landscape effectively. Third, as the depth734 increases, the performance improvement decreases. Deeper735 NNs provide more overconfident results (Guo et al., 2017),736 but the number of spatial smoothing layers calibrating uncer-737 tainty is fixed. Last, the performance improvement of ResNeXt, which includes an ensemble in its738 internal structure, is relatively marginal.739
Fig. E.1 shows predictive performance of MC dropout and deep ensemble for ensemble size. A740 deep ensemble with an ensemble size of 1 is a deterministic NN. This figure shows that spatial741 smoothing improves efficiency of ensemble size and the predictive performance at ensemble size of742
Table E.2: Spatial smoothing improves adversarial robustness. We measure the accuracy (ACC) and the Attack Success Rate (ASR) of ResNet-50 against adversarial attacks on ImageNet.
ATTACK MC DROPOUT SMOOTH ACC(%) ASR (%)
FGSM
· · 28.3 (+0.0) 62.9 (-0.0) · X 30.3 (+2.0) 60.5 (-2.4) X · 30.3 (+0.0) 59.8 (-0.0) X X 32.6 (+2.3) 57.4 (-2.4)
PGD
· · 7.5 (+0.0) 90.1 (-0.0) · X 9.0 (+1.4) 88.2 (-1.9) X · 12.2 (+0.0) 83.7 (-0.0) X X 13.7 (+1.5) 82.1 (-1.6)
50. In addition, spatial smoothing stabilizes NN training. It reduces the variance of the performance,743 especially in VGG.744
A peculiarity of the results on ImageNet is that spatial smoothing degrades ECE of ResNet-50. It745 is because spatial smoothing significantly improves the accuracy in this case, and there tends to be746 a trade-off between accuracy and ECE, e.g. as shown in (Guo et al., 2017), Fig. A.1, and Fig. B.3.747 Instead, spatial smoothing shows the improvement in NLL, another uncertainty metric.748
Predictive performance on training datasets. Note that spatial smoothing helps NN learn strong749 representations. In other words, spatial smoothing does not regularize NNs. For example, NLL750 ResNet-18 with MC dropout on CIFAR-100 training dataset is 2.20× 10−2. The NLL of the ResNet751 with spatial smoothing is 1.94× 10−2. In conclusion, spatial smoothing reduces the training loss.752
Corruption robustness. We measure predictive performance on CIFAR-100-C (Hendrycks &753 Dietterich, 2019) in order to evaluate the robustness of the models against 5 intensities and 15 types754 of data corruption. The top row of Fig. E.3 shows the results as a box plot. The box plot shows the755 median, interquartile range (IQR), minimum, and maximum of predictive performance for types.756 They reveal that spatial smoothing improves predictive performance for corrupted data. In particular,757 spatial smoothing undoubtedly helps in predicting reliable uncertainty.758
To summarize the performance of corrupted data in a single value, Hendrycks & Dietterich (2019) introduced a corruption error (CE) for quantitative comparison. CEfc , which is CE for corruption type c and model f , is as follows:
CEfc = ( 5∑ i=1 Efi,c )/( 5∑ i=1 EAlexNeti,c ) (31)
whereEfi,c is top-1 error of f for corruption type c and intensity i, andE AlexNet i,c is the error of AlexNet. Mean CE or mCE summarizes CEfc by averaging them over 15 corruption types such as Gaussian noise, brightness, and show. Likewise, to evaluate robustness in terms of uncertainty, we introduce corruption NLL (CNLL, ↓) and corruption ECE (CECE, ↓) as follows:
CNLLfc = ( 5∑ i=1 NLLfi,c )/( 5∑ i=1 NLLAlexNeti,c ) (32)
and
CECEfc = ( 5∑ i=1 ECEfi,c )/( 5∑ i=1 ECEAlexNeti,c ) (33)
Table E.3: Spatial smoothing improves the consistency, robustness against shift-perturbation. We measure the consistency of ResNet-18 on CIFAR-10-P. Deterministic NN with N = 5 means deep ensemble.
MC DROPOUT SMOOTH N CONS(%) CEC (×10−2)
· · 1 97.9 (+0.0) 1.03 (-0.00) · X 1 98.2 (+0.3) 1.16 (+0.13) · · 5 98.7 (+0.0) 1.22 (-0.00) · X 5 98.9 (+0.2) 1.33 (+0.11)

heading: X · 50 98.2 (+0.0) 1.29 (-0.00)
text: X X 50 98.4 (+0.2) 1.34 (+0.05)
where NLLfi,c and ECE f i,c are NLL and ECE of f for c and i, respectively. mCNLL and mCECE759 are averages over corruption types. Experimental results show that spatial smoothing improves the760 robustness against data corruption. See Fig. E.3 for the results.761
The bottom row of Fig. E.3 shows mCNLL, mCE, and mCECE for ensemble size. They indicates that762 spatial smoothing improves not only the efficiency but corruption robustness across a whole range of763 ensemble size.764
Adversarial robustness. We show that spatial smoothing also improves adversarial robustness.765 First, we measure the robustness, in terms of accuracy and attack success rate (ASR), of ResNet-766 50 on ImageNet against popular adversarial attacks, namely FGSM (Goodfellow et al., 2015) and767 PGD (Madry et al., 2018). Table E.2 indicate that both MC dropout and spatial smoothing improve768 robustness against adversarial attacks.769
Next, we find out how spatial smoothing improves adversarial robustness. To this end, similar770 to Section 2.2, we measure the accuracy on the test datasets with frequency-based adversarial771 perturbations. In this experiment, we use FGSM attack. This experimental result shows that spatial772 smoothing is particularly robust against high frequency (≥ 0.3π) adversarial attacks. This is because773 spatial smoothing is a low-pass filter, as we mentioned in Section 2.2. Since the ResNet is vulnerable774
−6 −4 −2 0 2 4 6 80
25
50
75
100
Re la
tiv e
co nf
id en
ce (%
)
Translation (px)
0.00 0.25 0.50 0.75 1.00
0.00
0.25
0.50
0.75
1.00
Deterministic Deterministic + Smooth
MC dropout MC dropout + Smooth
Figure E.4: Spatial smoothing improves the confidence when the predictions are incorrect. We define relative confidence (See Eq. (36)), and measure the metric of ResNet-18 on CIFAR10-P.
against high frequency adversarial attack, an ef-775 fective defense of spatial smoothing against high776 frequency attacks significantly improves the robust-777 ness.778
Consistency. To evaluate the translation invariance of models, we use consistency (Hendrycks & Dietterich, 2019; Zhang, 2019a), a metric representing translation consistency for shift-translated data sequences S = {x1, · · · ,xM+1}, as follows:
Consistency = 1
M M∑ i=1 1(g(xi) = g(xi+1))
(34)
where g(x) = arg max p(y|x,D). Table E.3 pro-779 vides consistency of ResNet-18 on CIFAR-10-P780 (Hendrycks & Dietterich, 2019). The results shows781 that MC dropout and deep ensemble improve con-782 sistency, and spatial smoothing improves consis-783 tency of both deterministic and Bayesian NNs.784
Prior works (Zhang, 2019a; Azulay & Weiss, 2019) investigated the fluctuation of predictive confidence
Table E.4: Spatial smoothing and temporal smoothing are complementary. We provide predictive performance of MC dropout in semantic segmentation on CamVid for each method. SPAT and TEMP each stand for spatial smoothing and temporal smoothing. CONS stands for consistency.
MC DROPOUT SPAT TEMP N NLL ACC(%) ECE (%)
CONS (%)
· · · 1 0.354 (+0.000) 92.3 (+0.0) 4.95 (+0.00) 95.1 (+0.0) · X · 1 0.318 (+0.036) 92.4 (+0.1) 4.54 (+0.41) 95.5 (+0.4) · · X 1 0.290 (+0.064) 92.5 (+0.2) 3.18 (+1.77) 96.3 (+1.2) · X X 1 0.278 (+0.076) 92.5 (+0.2) 3.03 (+1.92) 96.6 (+1.5) X · · 50 0.298 (+0.000) 92.5 (+0.0) 4.20 (+0.00) 95.4 (+0.0) X X · 50 0.284 (+0.014) 92.6 (+0.1) 3.96 (+0.24) 95.6 (+0.2) X · X 1 0.273 (+0.025) 92.6 (+0.1) 3.23 (+0.97) 96.4 (+1.0) X X X 1 0.260 (+0.038) 92.6 (+0.1) 2.71 (+1.49) 96.5 (+1.1)
on shift-translated data sequence. However, surprisingly, we find that confidence fluctuation has little to do with consistency. To demonstrate this claim, we introduce cross-entropy consistency (CEC, ↓), a metric that represents the fluctuation of confidence on a shift-translated data sequence S = {x1, · · · ,xM+1}, as follows:
CEC = − 1 M M∑ i=1 f(xi) · log(f(xi+1)) (35)
where f(x) = p(y|x,D). In Table E.3, high consistency does not mean low CEC; conversely, high785 consistency tends to be high CEC. Canonical NNs predict overconfident probabilities, and their786 confidence sometimes changes drastically from near-zero to near-one. Correspondingly, it results in787 low consistency but low CEC. On the contrary, well-calibrated NNs such as MC dropout provide788 confidence that oscillates between zero and one, which results in high CEC.789
To represent the NN reliability properly, we propose relative confidence (↑) as follows:
Relative confidence = p(ytrue|x,D) / max p(y|x,D) (36)
where max p(y|x,D) is confidence of predictive result and p(ytrue|x,D) is probability of the result790 for true label. It is 1 when NN classifies the image correctly, and less than 1 when NN classifies it791 incorrectly. Therefore, relative confidence is a metric that indicates the overconfidence of a prediction792 when NN’s prediction is incorrect.793
Figure E.4 shows a qualitative example of consistency on CIFAR-10-P by using relative confidence.794 This figure suggests that spatial smoothing improves consistency of both deterministic and Bayesian795 NN.796
E.2 SEMANTIC SEGMENTATION797
Table E.4 shows the performance of U-Net on the CamVid dataset. This table indicates that spatial798 smoothing improves accuracy, uncertainty, and consistency of deterministic and Bayesian NNs.799 This is consistent with the results in image classification. In addition, temporal smoothing leads800 to significant improvement in efficiency of ensemble size, accuracy, uncertainty, and consistency801 by exploiting temporal information. Moreover, temporal smoothing requires only one ensemble to802 achieve high predictive performance, since it cooperates with the temporally previous predictions. We803 obtain the best predictive and computational performance by using both temporal smoothing and804 spatial smoothing.805
F COMPARISON WITH ANTI-ALIASED CNN806
As we mentioned in Section 4, local means (Blur), also known as anti-aliased CNN (Zhang, 2019a),807 improve accuracy. Nevertheless, our work (Prob + Blur) has novelties in three respects: different808 motivation, improved uncertainty estimation, and analysis of how spatial smoothing works.809
Different motivation. The motivation of local means was to mitigate the aliasing effect of subsam-810 pling and to improve shift invariance. In contrast, our spatial smoothing is introduced to aggregate811 and ensemble nearby feature map points.812
Improved uncertainty estimation. We demonstrate that spatial smoothing improves not only813 accuracy, but also uncertainty estimation and robustness against natural corruptions and adversarial814 attacks all at the same time. Moreover, we show that spatial smoothing significantly enhances the815 performance of MC dropout. Since there typically tends to be a trade-off between accuracy and816 “uncertainty + robustness”—e.g. as shown in (Guo et al., 2017; Zhang et al., 2019; Geirhos et al.,817 2019; Zhang, 2019b), Fig. A.1, and Fig. B.3—in NN modeling, we believe our simple yet effective818 method makes major inroads into the uncertainty quantification and generalization.819
Analysis of how spatial smoothing improves performance. We find that the predictive perfor-820 mance improvement is not due to the anti-aliasing effect of local means.821
• Prob + Blur—our probabilistic spatial smoothing—improves the performance of pre-822 activation CNNs, but Blur alone—local mean or anti-aliased CNN—does not. In fact,823 contrary to (Zhang, 2019a), local mean degrades the predictive performance since it results824 in loss of information. It suggests that Prob plays an key role in prediction. For more details,825 see Appendix F.1.826
• Although the local filtering can result in loss of information, Zhang (2019a) experimentally827 observed an increase in both shift-invariance (as expected) and accuracy (which was be-828 yond expectation). However, “there exist a fundamental trade-off between ‘shift-invariance829 plus anti-aliasing’ and performance” (Zhang, 2019b). Moreover, it is difficult to relate830 anti-aliasing to improved uncertainty and robustness. Zhang (2019a) did not provide an831 explanation for these phenomena. As discussed in Appendix E.1, spatial smoothing helps832 NNs learn strong representations, not regularizes NNs.833
• Spatial smoothing is, surprisingly, robust against blur corruptions.834
We analyze how spatial smoothing improves predictive performance, by using loss landscape vi-835 sualization, Hessian eigenvalue spectra, and Fourier analysis. These analyzes draw the following836 conclusions:837
• Loss landscape visualization: Spatial smoothing stabilizes loss landscape fluc-838 tuations, caused by e.g. MC dropout. This results in stabilizing NN training839 and improving performance as well as generalization. See Figs. 8 and C.2.840 See also code/resources/losslandscapes/resnet_mcdo_18.gif and841 code/resources/losslandscapes/resnet_mcdo_smoothing_18.gif in the842 supplementary material.843
• Hessian eigenvalue spectra: Spatial smoothing suppresses outliers of Hessian eigenvalues,844 which disrupt NN training. See Figs. 7 and C.3.845
• Fourier analysis: Spatial smoothing effectively removes high frequency signals, including846 noise due to MC dropout. We also show that CNNs are vulnerable to high frequency noise847 and high frequency adversarial attacks. See Figs. 6 and D.2.848
We also provide theoretical analysis of how spatial smoothing works. We prove that dropout sharpens849 the loss landscape, and ensemble smoothens it. Since the spatial smoothing is a spatial ensemble, it850 significantly enhances the performance of MC dropout. See Appendix D.3 for more details. Further-851 more, we also show that training-phase ensemble significantly improves the predictive performance852 because it smoothens the loss landscape without loss of prediction diversity. Therefore, the spatial853 smoothing, which ensembles feature map points at training time, improves the performance effectively.854 See Appendix D.4.855
F.1 Prob PLAYS AN IMPORTANT ROLE IN SPATIAL SMOOTHING856
As discussed in Section 2.1, we take the perspective that each point in feature map is a prediction for857 binary classification by deriving the Bernoulli distributions from the feature map by using Prob. It is858 in contrast to previous works known as sampling-free BNNs (Hernández-Lobato & Adams, 2015;859 Wang et al., 2016; Wu et al., 2019) attempting to approximate the distribution of feature map with860 one Gaussian distribution. We do not use any assumptions on the distribution of feature map, and861 exactly represent the Bernoulli distributions and their averages. However, sampling-free BNNs are862 error-prone because there is no guarantee that feature maps will follow a Gaussian distribution.863
This Prob plays an important role in spatial smoothing. CNNs such as VGG, ResNet, and ResNeXt generally use post-activation arrangement. In other words, their stages end with BatchNorm and ReLU. Therefore, spatial smoothing layers Smooth(z) = Blur ◦ Prob(z) in CNNs cooperates with BatchNorm and ReLU as follows:
Prob(z) = ReLU ◦ tanhτ ◦ ReLU ◦ BatchNorm (z) (37) = ReLU ◦ tanhτ ◦ BatchNorm (z) (38)
since ReLU and tanhτ are commutative, and ReLU ◦ ReLU is ReLU. This Prob is trainable and is a864 general form of Eq. (7). If we only use Blur as spatial smoothing, the activations BatchNorm–ReLU865 play the role of Prob.866
100 101 Ensemble Size
1.00
1.20
1.40
1.60
1.80
N LL
100 101 Ensemble Size
1.00
1.20
1.40
1.60
1.80
N LL
Deterministic Deterministic + Blur Deterministic + Prob + Blur
MC dropout MC dropout + Blur MC dropout + Prob + Blur
Figure F.1: Blur alone harms the predictive performance, although Prob + Blur improves it. We provide NLL of pre-activation VGG-16 on CIFAR-100.
In order to analyze the roles of Prob and Blur867 more precisely, we measure the predictive perfor-868 mance of the model that does not use the post-869 activation. Figure F.1 shows NLL of pre-activation870 VGG-16 on CIFAR-100. The result shows that871 Blur with Prob improves the performance, but872 Blur alone does not. In fact, contrary to (Zhang,873 2019a), blur degrades the predictive performance874 since it results in loss of information. We also875 measure the performance of VGG-19, ResNet-18,876 ResNet-50, and BlurPool (Zhang, 2019a) with pre-877 activation, and observe the same phenomenon. In878 addition, BatchNorm–ReLU in front of GAP signif-879 icantly improves the performance of pre-activation880 ResNet.881
As mentioned in Appendix C.2, pre-activation is882 a special case of spatial smoothing. Therefore, the883 performance improvement of pre-activation by spa-884 tial smoothing is marginal compared to that of post-885 activation.886

Review summaries:
This paper proposed a spatial `smooth` layer including a feature range bounding layer `prob` and `blur` the intermediate feature map in a CNN. 'Smooth' improves the accuracy and uncertainty of both deterministic CNN and a Bayesian NN approximated by MC-dropout. Authors tried to justify how `smooth` improves the optimization of neural networks by 1. interpolating the `blur` operations as an ensemble of the neighboring features 2. showing `smooth` filter out the high-frequency noises introduced by MC-dropout and smoothen the loss landscapes perturbed by MC-dropout. Authors empirically evaluated `smooth` on image classification and semantic segmentation tasks and showed that it improves both accuracy and uncertainty. Authors also tried to connect common pieces in CNNs like global average pooling, ReLU + BN as special cases of `smooth`.

The motivation of this work is on the computational cost of using BNNs in practice, where applications might require a large number of BNNs in an ensemble formation for achieving good performance. The work in this manuscript aims to reduce the computational cost ensemble.  The manuscript's insight for solving the computational cost is to exploit similarities in spatial neighbouring from images. This is a similar approach used in convolutional neural networks. Experimental results show improved efficiency in image tasks, while also considerably reducing the computational cost when compared to competitor methods.