Blur Is an Ensemble: Spatial Smoothings to Improve Accuracy, Uncertainty, and Robustness

Bayesian neural networks ( BNNs ) have shown success in the areas of uncertainty1 estimation and robustness . However , a crucial challenge prohibits their use in2 practice . Bayesian NNs require a large number of predictions to produce reliable3 results , leading to a significant increase in computational cost . To alleviate this4 issue , we propose spatial smoothing , a method that ensembles neighboring feature5 map points of CNNs . By simply adding a few blur layers to the models , we6 empirically show that spatial smoothing improves accuracy , uncertainty estimation,7 and robustness of BNNs across a whole range of ensemble sizes . In particular,8 BNNs incorporating spatial smoothing achieve high predictive performance merely9 with a handful of ensembles . Moreover , this method also can be applied to canonical10 deterministic neural networks to improve the performances . A number of evidences11 suggest that the improvements can be attributed to the stabilized feature maps12 and the flattening of the loss landscape . In addition , we provide a fundamental13 explanation for prior works—namely , global average pooling , pre-activation , and14 ReLU6—by addressing them as special cases of spatial smoothing . These not15 only enhance accuracy , but also improve uncertainty estimation and robustness by16 making the loss landscape smoother in the same manner as spatial smoothing.17 1 INTRODUCTION18 Azulay & Weiss , 2019 ) .33 Bayesian neural networks ( BNNs ) , such as Monte Carlo34 ( MC ) dropout ( Gal & Ghahramani , 2016 ) , provide a35 probabilistic representation of NN weights . They com-36 bine a number of models selected based on weight prob-37 ability to make predictions of desired results . Thanks to38 this feature , BNNs have been widely used in the areas of uncertainty estimation ( Kendall & Gal , 2017 ) 39 and robustness ( Ovadia et al. , 2019 ) . They are also promising in other fields like out-of-distribution40 detection ( Malinin & Gales , 2018 ) and meta-learning ( Yoon et al. , 2018 ) .41 Nevertheless , there remains a significant challenge that prohibits their use in practice . BNNs require42 an ensemble size of up to fifty to achieve high predictive performance , which results in a fiftyfold43 increase in computational cost ( Kendall & Gal , 2017 ; Loquercio et al. , 2020 ) . Therefore , if BNNs44 can achieve high predictive performance merely with a handful of ensembles , they could be applied45 to a much wider range of areas.46 1.1 PRELIMINARY47 We would first like to discuss BNN inference in detail , then move on to Vector-Quantized BNN48 ( VQ-BNN ) inference ( Park et al. , 2021 ) , an efficient approximated BNN inference.49 BNN inference . Suppose we have access to posterior probability of NN weight p ( w|D ) for training dataset D. The predictive result of BNN is given by the following predictive distribution : p ( y|x0 , D ) = ∫ p ( y|x0 , w ) p ( w|D ) dw ( 1 ) wherex0 is observed input data vector , y is output vector , and p ( y|x , w ) is the probabilistic prediction parameterized by the result of NN for an input x and weight w. In most cases , the integral can not be solved analytically . Thus , we use the MC estimator to approximate it as follows : p ( y|x0 , D ) ' N−1∑ i=0 1 N p ( y|x0 , wi ) ( 2 ) wherewi ∼ p ( w|D ) and N is the number of the samples . The equation indicates that BNN inference50 is ensemble average of NN predictions for one observed data point as shown on the left of Fig . 2.51 Using N neural networks in the ensemble would requires N times more computational complexity52 than one NN execution.53 1.00 Data-complemented BNN inference . To reduce the computational cost of BNN inference , VQBNN ( Park et al. , 2021 ) executes NN for an observed data only once and complements the result with previously calculated predictions for other data . If we have access to previous predictions , the computational performance of VQ-BNN becomes comparable to that of one NN execution . To be specific , VQ-BNN inference is : p ( y|x0 , D ) ' N−1∑ i=0 π ( xi|x0 ) p ( y|xi , wi ) ( 3 ) where π ( xi|x0 ) is the importance of data xi with respect to the observed data x0 , and it is defined as a54 similarity betweenxi andx0 . p ( y|x0 , w0 ) is the newly calculated prediction , and { p ( y|x1 , w1 ) , · · · } 55 are previously calculated predictions . To accurately infer the results , the previous predictions should56 consist of predictions for “ data similar to the observed data ” .57 Thanks to the temporal consistency of real-world data streams , aggregating predictions for similar58 data in data streams is straightforward . Since temporally proximate data sequences tend to be similar,59 we can memorize recent predictions and calculates their average using exponentially decreasing60 importance . In other words , VQ-BNN inference for data streams is simply temporal smoothing of61 recent predictions as shown in the middle of Fig . 2.62 VQ-BNN has two limitations , although it may be a promising approach to obtain reliable results63 in an efficient way . First , it was only applicable to data streams such as video sequences . Applying64 VQ-BNN to images is challenging because it is impossible to memorize all similar images in advance.65 Second , Park et al . ( 2021 ) used VQ-BNN only in the testing phase , not in the training phase . We find66 that ensembling predictions for similar data helps in NN training by smoothing the loss landscape.67 1.2 MAIN CONTRIBUTION68 1 Spatially neighboring points in visual imagery tend to be similar , as do feature maps of convolu-69 tional neural networks ( CNNs ) . By exploiting this spatial consistency , we propose spatial smoothing70 as a method of ensembling nearby feature maps to improve the efficiency of ensemble size in BNN71 inference . The right side of Fig . 2 visualizes spatial smoothing aggregating neighboring feature maps.72 2 We empirically demonstrate that spatial smoothing improves the efficiency in vision tasks , such73 as image classification on CIFAR ( Krizhevsky et al. , 2009 ) and ImageNet ( Russakovsky et al. , 2015 ) ,74 without any additional training parameters . Figure 3 shows that negative log-likelihood ( NLL ) of75 “ MC dropout + spatial smoothing ” with an ensemble size of two is comparable to that of vanilla MC76 dropout with an ensemble size of fifty . We also demonstrate that spatial smoothing improves accuracy,77 uncertainty , and robustness all at the same time . Figure 1 shows that spatial smoothing improves both78 the accuracy and uncertainty of various deterministic and Bayesian NNs with an ensemble size of79 fifty on CIFAR-100.80 3 Global average pooling ( GAP ) ( Lin et al. , 2014 ; Zhou et al. , 2016 ) , pre-activation ( He et al.,81 2016b ) , and ReLU6 ( Krizhevsky & Hinton , 2010 ; Sandler et al. , 2018 ) have been widely used in vision82 tasks . However , their motives are largely justified by the experiments . We provide an explanation for83 these methods by addressing them as special cases of spatial smoothing . Experiments support the84 claim by showing that the methods improve not only accuracy but also uncertainty and robustness.85 2 PROBABILISTIC SPATIAL SMOOTHING86 To improve the computational perfor-87 mance of BNN inference , VQ-BNN88 ( Park et al. , 2021 ) executes NN pre-89 diction only once and complements90 the result with previously calculated91 predictions . The key to the success92 of this approach largely depends on93 the collection of previous predictions94 for proximate data . Gathering tempo-95 rally proximate data and their predic-96 tions from data streams is easy be-97 cause recent data and predictions can98 be aggregated using temporal consis-99 tency . On the other hand , gathering100 time-independent proximate data , e.g . images , is more difficult because they lack such consistency.101 2.1 MODULE ARCHITECTURE FOR ENSEMBLING NEIGHBORING FEATURE MAP POINTS102 So instead of temporal consistency , we use spatial consistency—where neighboring pixels of images103 are similar—for real-world images . Under this hypothesis , we take the feature maps as predictions104 and aggregate neighboring feature maps.105 Most CNN architectures , including ResNet , consist of multiple stages that begin with increasing the number of channels while reducing the spatial dimension of the input volume . We decompose an entire BNN inference into several steps by rewriting each stage in a recurrence relation as follows : p ( zi+1|zi , D ) = ∫ p ( zi+1|zi , wi ) p ( wi|D ) dwi ( 4 ) where zi is input volume of the i-th stage , and the first and the last volume are input data and106 output . wi and p ( wi|D ) are NN weight in the i-th stage and its probability . p ( zi+1|zi , wi ) is output107 probability of zi+1 with respect to the input volume zi . To derive the probability from the output108 feature map , we transform each point of the feature map into a Bernoulli distribution . To do so , a109 composition of tanh and ReLU , a function from value of range [ −∞ , ∞ ] into probability , is added110 after each stage . Put shortly , we use neural networks for point-wise binary feature classification.111 Since Eq . ( 4 ) is a kind of BNN inference , it can be approximated using Eq . ( 3 ) . In other words , each112 stage predicts feature map points only once and complements predictions with similar feature maps.113 Under spatial consistency , it averages probabilities of spatially neighboring feature map points , which114 is well known as blur operation in image processing . For the sake of implementation simplicity,115 average pooling with a kernel size of 2 and a stride of 1 is used as a box blur . This operation ensembles116 four neighboring probabilities with the same importances.117 In summary , as shown in Fig . 4 , we propose the following probabilistic spatial smoothing layer : Smooth ( z ) = Blur ◦ Prob ( z ) ( 5 ) where Prob ( · ) is a point-wise function from a feature map to probability , and Blur ( · ) is importance-118 weighted average for ensembling spatially neighboring probabilities from feature maps . Smooth layer119 is added after each stage . Prob and Blur are further elaborated below.120 Prob : Feature map to probability . Prob is a function that transforms a real-valued feature map into probability . We use tanh–ReLU composition for this purpose . However , tanh is commonly known to suffer from the vanishing gradient problem . To alleviate this issue , we propose the following temperature-scaled tanh : tanhτ ( z ) = τ tanh ( z/τ ) ( 6 ) where τ is a hyperparameter called temperature . τ is 1 in conventional tanh and ∞ in identity121 function . tanhτ imposes an upper bound on a value , but does not limit the upper bound to 1.122 MC dropout MC dropout + Smooth . An unnormalized probability , ranging from 0 to τ , is allowed as the output of Prob . Then , thanks to the linearity of integration , we obtain an unnormalized predictive distribution accordingly . Taking this into account , we propose the following Prob : Prob ( z ) = ReLU ◦ tanhτ ( z ) ( 7 ) where τ > 1 . We empirically determine τ to minimize NLL , a metric that measures both accuracy123 and uncertainty . See Fig . B.3 for more detailed ablation studies . In addition , we expect upper-bounded124 functions , e.g. , ReLU6 ( z ) = ReLU ◦ min ( z , 6 ) and feature map scaling z/τ with τ > 1 which125 is BatchNorm , to be able to replace tanhτ in Prob ; and as expected , these alternatives improve126 uncertainty estimation in addition to accuracy . See Appendix C.2 and Appendix C.3 for detailed127 discussions on activation ( ReLU ◦ BatchNorm ) and ReLU6 as Prob.128 Blur : Averaging neighboring probabilities . Blur averages the probabilities from feature maps . We primarily use the average pool with a kernel size of 2 and a stride of 1 as the implementation of Blur for the sake of simplicity . Nevertheless , we could generalize Blur by using the following depth-wise convolution , which acts on each input channel separately , with non-trainable kernel K = 1 ||k||21 k ⊗ k > ( 8 ) where k is a 1D matrix , e.g. , k ∈ { ( 1 ) , ( 1 , 1 ) , ( 1 , 2 , 1 ) , ( 1 , 4 , 6 , 4 , 1 ) } . Different ks derive different129 importances for neighboring feature maps . We empirically show that most Blurs improve the130 predictive performance and that optimalK varies by model . For more ablation studies , see Table B.2.131 2.2 HOW DOES SPATIAL SMOOTHING HELP OPTIMIZATION ? 132 We present theoretical and empirical aspects to show that spatial smoothing ensembles feature maps.133 Feature map variance . BNNs have two types of uncertainties : One is model uncertainty and the134 other is data uncertainty ( Park et al. , 2021 ) . These randomnesses increase the variance of the feature135 maps . To demonstrate that spatial smoothing is an ensemble , we use the following proposition:136 Proposition 1 . Ensembles reduce the variance of predictions.137 We omit the proof since it is straightforward . In our context , predictions are output feature maps of a138 stage . We investigate model and data uncertainties of the predictions along NN layers to show that139 spatial smoothing reduces the randomnesses and ensembles feature maps . Figure 5 shows the model140 uncertainty and data uncertainty of Bayesian ResNet including MC dropout layers . In this figure , the141 uncertainty of MC dropout ’ s feature map only accumulates , and almost monotonically increases in142 every NN layer . In contrast , the uncertainty of “ MC dropout + spatial smoothing ” ’ s feature map is143 significantly decreases at the end of stages , suggesting that the smoothing layers ensemble the feature144 map . In other words , they make the feature map more accurate and stabilized input volumes for the145 next stages . In addition , consistently , the spatial smoothing layer close to the last layer significantly146 improves performance because it reduces the uncertainty of predictions largely . See Fig . B.5 for more147 detailed results . Deterministic NNs do not have model uncertainty but data uncertainty . Therefore,148 spatial smoothing improves the performance of deterministic NNs as well as Bayesian NNs.149 Fourier analysis . We also analyze spatial smoothing through the lens of Fourier transform:150 Proposition 2 . Ensembles filter high-frequency signals.151 The proof is provided in Eqs . ( 16 ) to ( 17 ) . Figure 6b shows the 2D Fourier transformed output152 feature map at the end of the stage 1 . This figure reveals that MC dropout almost does not affect153 low-frequency ( < 0.3π ) ranges , and it adds high-frequency ( ≥ 0.3π ) noises . Since spatial smoothing154 is a low-pass filter , it effectively filters high-frequency signals , including the noises caused by MC155 dropout.156 We also find that CNNs are particularly vulnerable to high-frequency noises . To demonstrate this157 claim , following Shao et al . ( 2021 ) , we measure accuracy with respect to data with frequency-based158 random noise xnoise = x0 +F−1 ( F ( δ ) Mf ) , where x0 is clean data , F ( · ) and F−1 ( · ) are Fourier159 transform and inverse Fourier transform , δ is random noise , and Mf is frequency mask as shown160 in Fig . 6a . Figure 6c exhibits the results . In sum , high-frequency noises , including those caused by161 MC dropout , significantly impair accuracy . Spatial smoothing improves the robustness by effectively162 removing high-frequency noises.163 Loss landscape . Lastly , we show that the randomness hinders NN training as follows:164 Proposition 3 . Randomness of predictions sharpens the loss landscape , and ensembles flatten it.165 The proof is provided in Eqs . ( 18 ) to ( 25 ) . Since a sharp loss function disturbs NN optimization166 ( Keskar et al. , 2017 ; Santurkar et al. , 2018 ; Foret et al. , 2020 ) , reducing the uncertainty helps NN167 learn strong representations . For example , training phase NN ensemble averages out the randomness,168 and it flattens the loss function . In consequence , an ensemble of BNN outputs in training phase169 significantly improves the predictive performance . See Fig . D.4 for numerical results . However , we170 do not use training phase ensemble because it significantly increases the training time . Instead , we171 use spatial smoothing as a method that ensembles feature maps without sacrificing training time.172 We visualizes the loss landscapes ( Li et al. , 2018 ) , the contours of NLL on training dataset . Figure 8b173 shows that the loss landscapes of MC dropout fluctuate and have irregular surfaces due to the174 randomness . As Li et al . ( 2018 ) ; Foret et al . ( 2020 ) pointed out,175 this may lead to poor generalization and predictive performance.176 Spatial smoothing reduces randomness as discussed above , and177 spatial smoothing aids in optimization by stabilizing and flattening178 the loss landscape of BNN as shown in Fig . 8c.179 Furthermore , we use Hessian to quantitatively represent the sharp-180 ness of the loss landscapes . Figure 7 shows the Hessian max eigen-181 value spectra of the models in Fig . 8 with a batch size of 128 , which182 reveals that spatial smoothing reduces the magnitude of Hessian183 eigenvalues and suppresses outliers . Since large Hessian eigenval-184 ues disturb NN training ( Ghorbani et al. , 2019 ) , we come to the185 same conclusion that spatial smoothing helps NN optimization . See186 Appendix C.1 for a more detailed description of the configurations187 of the Hessian max eigenvalue spectra . In addition , from these188 observations , we propose the conjecture that the flatter the loss189 landscape , the better the uncertainty estimation , and vice versa.190 2.3 REVISITING GLOBAL AVERAGE POOLING191 The success of GAP classifier in image classification is192 indisputable . The initial motivation and the most widely193 accepted explanation for this success is that GAP prevents194 overfitting by using far fewer parameters than multi-layer195 perceptron ( MLP ) ( Lin et al. , 2014 ) . However , we discover196 that the explanation is poorly supported . We compares197 GAP with other classifiers including MLP . Contrary to198 popular belief , Table 1 suggests that MLP does not overfit199 the training dataset . MLP underfits or gives comparable200 performance to GAP on the training dataset . On the test201 dataset , GAP provides better results compared with MLP . See Table C.1 for more detailed results.202 Our argument is that GAP is an extreme case of spatial smoothing . In other words , GAP is successful203 because it ensembles feature maps and smoothens the loss landscape to help optimization . To support204 this claim , we visualizes the loss landscape of MLP as shown in Fig . 8a . It is chaotic compared to205 that of GAP as shown in Fig . 8b . Hessian shows the consistent results as demonstrated by Fig . 7.206 3 EXPERIMENTS207 This section presents two experiments . The first experiment is image classification through which208 we show that spatial smoothing not only improves the ensemble efficiency , but also the accuracy,209 uncertainty , and robustness of both deterministic NN and MC dropout . The second experiment is210 semantic segmentation on data streams through which we show that spatial smoothing and temporal211 smoothing ( Park et al. , 2021 ) are complementary . See Appendix A for more detailed configurations.212 1.00 Three metrics are measured in these experiments : NLL ( ↓1 ) , accuracy ( ↑ ) , and expected calibration213 error ( ECE , ↓ ) ( Guo et al. , 2017 ) . NLL represents both accuracy and uncertainty , and is the most214 widely used as a proper scoring rule . ECE measures discrepancy between accuracy and confidence.215 3.1 IMAGE CLASSIFICATION216 This section mainly discuss ResNet ( He et al. , 2016a ) . Table E.1 also discuss other settings that217 show the same trend : e.g. , VGG ( Simonyan & Zisserman , 2015 ) , ResNeXt ( Xie et al. , 2017 ) ,218 and pre-activation models ( He et al. , 2016a ) . Spatial smoothing also improves deep ensemble219 ( Lakshminarayanan et al. , 2017 ) , another non-Bayesian probabilistic NN method . See Fig . E.1.220 Performance . Fig . 3 and Fig . 9 show the predictive performances of ResNet-18 on CIFAR-100221 and ResNet-50 on ImageNet , respectively . The results indicate that spatial smoothing improves both222 accuracy and uncertainty in many respects . Let us be more specific . First , spatial smoothing improves223 the efficiency of ensemble size . In these examples , the NLL of “ MC dropout + spatial smoothing ” 224 with an ensemble size of 2 is comparable to or even better than that of MC dropout with an ensemble225 size of 50 . In other words , “ MC dropout + spatial smoothing ” is 25× faster than MC dropout with226 a similar predictive performance . Second , the predictive performance of “ MC dropout + spatial227 smoothing ” is better than that of MC dropout , at an ensemble size of 50 . Third , spatial smoothing228 improves the predictive performance of deterministic NN , as well as MC dropout.229 Robustness . To evaluate robustness against data corruption , we230 measure predictive performance of ResNet-18 on CIFAR-100-231 C ( Hendrycks & Dietterich , 2019 ) . This dataset consists of data232 corrupted by 15 different types , each with 5 levels of intensity233 each . We use mean corruption NLL ( mCNLL , ↓ ) , the averages234 of NLL over intensities and corruption types , to summarize the235 performance of corrupted data in a single value . See Eq . ( 32 ) for236 a more rigorous definition . Figure 10 shows that spatial smoothing237 not only improves the efficiency but also corruption robustness238 across a whole range of ensemble size . See Fig . E.3 for more239 details . Spatial smoothing also improves adversarial robustness240 and perturbation consistency ( ↑ ) ( Hendrycks & Dietterich , 2019 ; 241 Zhang , 2019a ) , shift-transformation invariance . See Table E.2,242 Table E.3 , and Fig . E.4 for more details.243 3.2 SEMANTIC SEGMENTATION244 Table 2 summarizes the result of semantic segmentation on CamVid dataset ( Brostow et al. , 2008 ) 245 that consists of real-world 360×480 pixels videos . The table shows that spatial smoothing improves246 predictive performance , which is consistent with the image classification experiment . Moreover , the247 result reveals that spatial smoothing and temporal smoothing ( Park et al. , 2021 ) are complementary.248 See Table E.4 for more results.249 1We use arrows to indicate which direction is better . Spatial smoothing can be compared with prior works in the following areas.251 Anti-aliased CNNs . Local means ( Zhang , 2019a ; Zou et al. , 2020 ; Vasconcelos et al. , 2020 ; Sinha252 et al. , 2020 ) were introduced for the shift-invariance of deterministic CNNs in image classification.253 They were motivated to prevent the aliasing effect of subsampling . Although the local filtering can254 result in a loss of information , Zhang ( 2019a ) experimentally observed an increase in accuracy that255 was beyond expectation . We provide a fundamental explanation for this phenomenon : Local means256 are a spatial ensemble . An ensemble not only improves accuracy , but also uncertainty and robustness257 of deterministic and Bayesian NNs . In Fig . F.1 , we also show that the predictive performance258 improvement is not due to anti-aliasing of local mean . See Appendix F for more discussion on local259 means . For a discussion on non-local means ( Wang et al. , 2018 ) and self-attention ( Dosovitskiy et al.,260 2021 ) , see Section 5.261 Sampling-free BNNs . Sampling-free BNNs ( Hernández-Lobato & Adams , 2015 ; Wang et al.,262 2016 ; Wu et al. , 2019 ) predict results based on a single or couple of NN executions . To this end , it is263 assumed that posterior and feature maps follow Gaussian distributions . However , the discrepancy264 between reality and assumption accumulates in every NN layer . Consequently , to the best of our265 knowledge , most of the sampling-free BNNs could only be applied to shallow models , such as LeNet,266 and were tested on small datasets . Postels et al . ( 2019 ) applied sampling-free BNNs to SegNet ; 267 nonetheless , Park et al . ( 2021 ) argued that they do not predict well-calibrated results.268 Efficient deep ensembles . Deep ensemble ( Lakshminarayanan et al. , 2017 ; Fort et al. , 2019 ) is269 another probabilistic NN approach for predicting reliable results . BatchEnsemble ( Wen et al. , 2020 ; 270 Dusenberry et al. , 2020 ) ensembles over a low-rank subspace to make deep ensemble more efficient.271 Depth uncertainty network ( Antoran et al. , 2020 ) aggregates feature maps from different depths of272 a single NN to predict results efficiently . Despite being robust against data corruption , it provides273 weaker predictive performance compared to deterministic NN and MC dropout.274 5 DISCUSSION275 We propose spatial smoothing , a simple yet efficient module to improve BNN . Three different per-276 spectives , namely , feature map variance , Fourier analysis , and loss landscape , suggest that spatial277 smoothing ensembles feature maps . The limitation of spatial smoothing is that designing its compo-278 nents requires inductive bias . In other words , the optimal shape of the blur kernel is model-dependent.279 We believe this problem can be solved by introducing self-attention ( Vaswani et al. , 2017 ) . Self-280 attentions for computer vision ( Dosovitskiy et al. , 2021 ; Touvron et al. , 2021 ; Carion et al. , 2020 ) 281 can be deemed as trainable importance-weighted ensembles of feature maps . The observation that282 Transformers are more robust than expected ( Bhojanapalli et al. , 2021 ; Shao et al. , 2021 ) supports this283 claim . Therefore , using self-attentions to generalize spatial smoothing would be a promising future284 work because it not only expands our work , but also helps deepen our understanding of self-attention.285 REPRODUCIBILITY STATEMENT286 To ensure reproducibility , we provide comprehensive resources , such as code and experimental details.287 The codebase will be released as open source under the Apache License 2.0 . See the supplemental288 material for the code . Appendix A provides the specifications of all models used in this work . Detailed289 experimental setup including hyperparameters and ablation study are also available in Appendix A290 and Appendix B. De-facto image datasets are used for all experiments as described in Appendix A.291 REFERENCES292 Javier Antoran , James Allingham , and José Miguel Hernández-Lobato . Depth uncertainty in neural293 networks . Advances in Neural Information Processing Systems , 2020.294 Aharon Azulay and Yair Weiss . Why do deep convolutional networks generalize so poorly to small295 image transformations ? Journal of Machine Learning Research , 2019.296 Srinadh Bhojanapalli , Ayan Chakrabarti , Daniel Glasner , Daliang Li , Thomas Unterthiner , and297 Andreas Veit . Understanding robustness of transformers for image classification . In Proceedings298 of the IEEE/CVF International Conference on Computer Vision , 2021.299 Gabriel J Brostow , Jamie Shotton , Julien Fauqueur , and Roberto Cipolla . Segmentation and recog-300 nition using structure from motion point clouds . In European Conference on Computer Vision.301 Springer , 2008.302 Nicolas Carion , Francisco Massa , Gabriel Synnaeve , Nicolas Usunier , Alexander Kirillov , and Sergey303 Zagoruyko . End-to-end object detection with transformers . In European Conference on Computer304 Vision . Springer , 2020.305 Alexey Dosovitskiy , Lucas Beyer , Alexander Kolesnikov , Dirk Weissenborn , Xiaohua Zhai , Thomas306 Unterthiner , Mostafa Dehghani , Matthias Minderer , Georg Heigold , Sylvain Gelly , et al . An image307 is worth 16x16 words : Transformers for image recognition at scale . In International Conference308 on Learning Representations , 2021.309 Michael Dusenberry , Ghassen Jerfel , Yeming Wen , Yian Ma , Jasper Snoek , Katherine Heller , Balaji310 Lakshminarayanan , and Dustin Tran . Efficient and scalable bayesian neural nets with rank-1311 factors . In International Conference on Machine Learning . PMLR , 2020.312 Logan Engstrom , Brandon Tran , Dimitris Tsipras , Ludwig Schmidt , and Aleksander Madry . Exploring313 the landscape of spatial robustness . In International Conference on Machine Learning . PMLR,314 2019.315 Pierre Foret , Ariel Kleiner , Hossein Mobahi , and Behnam Neyshabur . Sharpness-aware minimization316 for efficiently improving generalization . In International Conference on Learning Representations,317 2020.318 Stanislav Fort , Huiyi Hu , and Balaji Lakshminarayanan . Deep ensembles : A loss landscape perspec-319 tive . arXiv preprint arXiv:1912.02757 , 2019.320 Jonathan Frankle , David J Schwab , and Ari S Morcos . Training batchnorm and only batchnorm:321 On the expressive power of random features in cnns . In International Conference on Learning322 Representations , 2021.323 Yarin Gal and Zoubin Ghahramani . Dropout as a bayesian approximation : Representing model324 uncertainty in deep learning . In International Conference on Machine Learning . PMLR , 2016.325 Robert Geirhos , Patricia Rubisch , Claudio Michaelis , Matthias Bethge , Felix A Wichmann , and326 Wieland Brendel . Imagenet-trained cnns are biased towards texture ; increasing shape bias improves327 accuracy and robustness . In International Conference on Learning Representations , 2019.328 Behrooz Ghorbani , Shankar Krishnan , and Ying Xiao . An investigation into neural net optimization329 via hessian eigenvalue density . In International Conference on Machine Learning . PMLR , 2019.330 Ian J Goodfellow , Jonathon Shlens , and Christian Szegedy . Explaining and harnessing adversarial331 examples . International Conference on Learning Representations , 2015.332 Priya Goyal , Piotr Dollár , Ross Girshick , Pieter Noordhuis , Lukasz Wesolowski , Aapo Kyrola,333 Andrew Tulloch , Yangqing Jia , and Kaiming He . Accurate , large minibatch sgd : Training imagenet334 in 1 hour . arXiv preprint arXiv:1706.02677 , 2017.335 Chuan Guo , Geoff Pleiss , Yu Sun , and Kilian Q Weinberger . On calibration of modern neural336 networks . In International Conference on Machine Learning . PMLR , 2017.337 Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Deep residual learning for image338 recognition . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,339 2016a.340 Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Identity mappings in deep residual341 networks . In European Conference on Computer Vision . Springer , 2016b.342 Dan Hendrycks and Thomas Dietterich . Benchmarking neural network robustness to common343 corruptions and perturbations . In International Conference on Learning Representations , 2019.344 José Miguel Hernández-Lobato and Ryan Adams . Probabilistic backpropagation for scalable learning345 of bayesian neural networks . In International Conference on Machine Learning . PMLR , 2015.346 Elad Hoffer , Tal Ben-Nun , Itay Hubara , Niv Giladi , Torsten Hoefler , and Daniel Soudry . Augment347 your batch : Improving generalization through instance repetition . In Proceedings of the IEEE/CVF348 Conference on Computer Vision and Pattern Recognition , 2020.349 A Kendall , V Badrinarayanan , and R Cipolla . Bayesian segnet : Model uncertainty in deep convolu-350 tional encoder-decoder architectures for scene understanding . In BMVC , 2017.351 Alex Kendall and Yarin Gal . What uncertainties do we need in bayesian deep learning for computer352 vision ? Advances in Neural Information Processing Systems , 2017.353 Nitish Shirish Keskar , Dheevatsa Mudigere , Jorge Nocedal , Mikhail Smelyanskiy , and Ping Tak Peter354 Tang . On large-batch training for deep learning : Generalization gap and sharp minima . In355 International Conference on Learning Representations , 2017.356 Alex Krizhevsky and Geoff Hinton . Convolutional deep belief networks on cifar-10 . Unpublished357 manuscript , 2010.358 Alex Krizhevsky , Geoffrey Hinton , et al . Learning multiple layers of features from tiny images . 2009.359 Balaji Lakshminarayanan , Alexander Pritzel , and Charles Blundell . Simple and scalable predictive360 uncertainty estimation using deep ensembles . In Advances in Neural Information Processing361 Systems , 2017.362 Hao Li , Zheng Xu , Gavin Taylor , Christoph Studer , and Tom Goldstein . Visualizing the loss landscape363 of neural nets . In Advances in Neural Information Processing Systems , 2018.364 Min Lin , Qiang Chen , and Shuicheng Yan . Network in network . In International Conference on365 Learning Representations , 2014.366 Antonio Loquercio , Mattia Segu , and Davide Scaramuzza . A general framework for uncertainty367 estimation in deep learning . IEEE Robotics and Automation Letters , 2020.368 Aleksander Madry , Aleksandar Makelov , Ludwig Schmidt , Dimitris Tsipras , and Adrian Vladu.369 Towards deep learning models resistant to adversarial attacks . In International Conference on370 Learning Representations , 2018.371 A Malinin and M Gales . Predictive uncertainty estimation via prior networks . In Advances in Neural372 Information Processing Systems . Curran Associates , Inc. , 2018.373 Yaniv Ovadia , Emily Fertig , Jie Ren , Zachary Nado , David Sculley , Sebastian Nowozin , Joshua374 Dillon , Balaji Lakshminarayanan , and Jasper Snoek . Can you trust your model ’ s uncertainty ? 375 evaluating predictive uncertainty under dataset shift . In Advances in Neural Information Processing376 Systems , 2019.377 Namuk Park , Taekyu Lee , and Songkuk Kim . Vector quantized bayesian neural network inference378 for data streams . In AAAI Conference on Artificial Intelligence , 2021.379 Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor380 Killeen , Zeming Lin , Natalia Gimelshein , Luca Antiga , et al . Pytorch : An imperative style,381 high-performance deep learning library . Advances in Neural Information Processing Systems,382 2019.383 Janis Postels , Francesco Ferroni , Huseyin Coskun , Nassir Navab , and Federico Tombari . Sampling-384 free epistemic uncertainty estimation using approximated variance propagation . In Proceedings of385 the IEEE/CVF International Conference on Computer Vision , 2019.386 Olaf Ronneberger , Philipp Fischer , and Thomas Brox . U-net : Convolutional networks for biomedical387 image segmentation . In International Conference on Medical image computing and computer-388 assisted intervention . Springer , 2015.389 Olga Russakovsky , Jia Deng , Hao Su , Jonathan Krause , Sanjeev Satheesh , Sean Ma , Zhiheng Huang,390 Andrej Karpathy , Aditya Khosla , Michael Bernstein , Alexander C. Berg , and Li Fei-Fei . ImageNet391 Large Scale Visual Recognition Challenge . International Journal of Computer Vision , pp . 211–252,392 2015.393 Mark Sandler , Andrew Howard , Menglong Zhu , Andrey Zhmoginov , and Liang-Chieh Chen . Mo-394 bilenetv2 : Inverted residuals and linear bottlenecks . In Proceedings of the IEEE Conference on395 Computer Vision and Pattern Recognition , 2018.396 Shibani Santurkar , Dimitris Tsipras , Andrew Ilyas , and Aleksander Madry . How does batch normal-397 ization help optimization ? Advances in Neural Information Processing Systems , 2018.398 Rulin Shao , Zhouxing Shi , Jinfeng Yi , Pin-Yu Chen , and Cho-Jui Hsieh . On the adversarial robustness399 of visual transformers . arXiv preprint arXiv:2103.15670 , 2021.400 Karen Simonyan and Andrew Zisserman . Very deep convolutional networks for large-scale image401 recognition . In International Conference on Learning Representations , 2015.402 Samarth Sinha , Animesh Garg , and Hugo Larochelle . Curriculum by smoothing . Advances in Neural403 Information Processing Systems , 2020.404 Hugo Touvron , Matthieu Cord , Matthijs Douze , Francisco Massa , Alexandre Sablayrolles , and Hervé405 Jégou . Training data-efficient image transformers & distillation through attention . In International406 Conference on Machine Learning . PMLR , 2021.407 Cristina Vasconcelos , Hugo Larochelle , Vincent Dumoulin , Nicolas Le Roux , and Ross Goroshin.408 An effective anti-aliasing approach for residual networks . arXiv preprint arXiv:2011.10675 , 2020.409 Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N Gomez , Łukasz410 Kaiser , and Illia Polosukhin . Attention is all you need . In Advances in Neural Information411 Processing Systems , 2017.412 Hao Wang , Xingjian Shi , and Dit-Yan Yeung . Natural-parameter networks : A class of probabilistic413 neural networks . Advances in Neural Information Processing Systems , 2016.414 Xiaolong Wang , Ross Girshick , Abhinav Gupta , and Kaiming He . Non-local neural networks . In415 Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018.416 Yeming Wen , Dustin Tran , and Jimmy Ba . Batchensemble : an alternative approach to efficient417 ensemble and lifelong learning . In International Conference on Learning Representations , 2020.418 Anqi Wu , Sebastian Nowozin , Edward Meeds , Richard E Turner , José Miguel Hernández-Lobato,419 and Alexander L Gaunt . Deterministic variational inference for robust bayesian neural networks.420 In International Conference on Learning Representations , 2019.421 Saining Xie , Ross Girshick , Piotr Dollár , Zhuowen Tu , and Kaiming He . Aggregated residual422 transformations for deep neural networks . In Proceedings of the IEEE Conference on Computer423 Vision and Pattern Recognition , 2017.424 Zhewei Yao , Amir Gholami , Kurt Keutzer , and Michael W Mahoney . Pyhessian : Neural networks425 through the lens of the hessian . In 2020 IEEE International Conference on Big Data ( Big Data ) .426 IEEE , 2020.427 Jaesik Yoon , Taesup Kim , Ousmane Dia , Sungwoong Kim , Yoshua Bengio , and Sungjin Ahn.428 Bayesian model-agnostic meta-learning . In Advances in Neural Information Processing Systems,429 2018.430 Sergey Zagoruyko and Nikos Komodakis . Wide residual networks . In BMVC , 2016.431 Hongyang Zhang , Yaodong Yu , Jiantao Jiao , Eric Xing , Laurent El Ghaoui , and Michael Jordan.432 Theoretically principled trade-off between robustness and accuracy . In International Conference433 on Machine Learning . PMLR , 2019.434 Richard Zhang . Making convolutional networks shift-invariant again . In International Conference on435 Machine Learning . PMLR , 2019a.436 Richard Zhang . Official meta-review of making convolutional networks shift-invariant again , 2019b.437 URL https : //openreview.net/forum ? id=SklVEnR5K7 & noteId=rklZnFS-gN.438 Bolei Zhou , Aditya Khosla , Agata Lapedriza , Aude Oliva , and Antonio Torralba . Learning deep439 features for discriminative localization . In Proceedings of the IEEE Conference on Computer440 Vision and Pattern Recognition , 2016.441 Xueyan Zou , Fanyi Xiao , Zhiding Yu , and Yong Jae Lee . Delving deeper into anti-aliasing in convnets.442 In BMVC , 2020.443

Summary: This paper proposed a spatial `smooth` layer including a feature range bounding layer `prob` and `blur` the intermediate feature map in a CNN. 'Smooth' improves the accuracy and uncertainty of both deterministic CNN and a Bayesian NN approximated by MC-dropout. Authors tried to justify how `smooth` improves the optimization of neural networks by 1. interpolating the `blur` operations as an ensemble of the neighboring features 2. showing `smooth` filter out the high-frequency noises introduced by MC-dropout and smoothen the loss landscapes perturbed by MC-dropout. Authors empirically evaluated `smooth` on image classification and semantic segmentation tasks and showed that it improves both accuracy and uncertainty. Authors also tried to connect common pieces in CNNs like global average pooling, ReLU + BN as special cases of `smooth`.