Blur Is an Ensemble: Spatial Smoothings to Improve Accuracy, Uncertainty, and Robustness

Bayesian neural networks ( BNNs ) have shown success in the areas of uncertainty1 estimation and robustness . However , a crucial challenge prohibits their use in2 practice . Bayesian NNs require a large number of predictions to produce reliable3 results , leading to a significant increase in computational cost . To alleviate this4 issue , we propose spatial smoothing , a method that ensembles neighboring feature5 map points of CNNs . By simply adding a few blur layers to the models , we6 empirically show that spatial smoothing improves accuracy , uncertainty estimation,7 and robustness of BNNs across a whole range of ensemble sizes . In particular,8 BNNs incorporating spatial smoothing achieve high predictive performance merely9 with a handful of ensembles . Moreover , this method also can be applied to canonical10 deterministic neural networks to improve the performances . A number of evidences11 suggest that the improvements can be attributed to the stabilized feature maps12 and the flattening of the loss landscape . In addition , we provide a fundamental13 explanation for prior works—namely , global average pooling , pre-activation , and14 ReLU6—by addressing them as special cases of spatial smoothing . These not15 only enhance accuracy , but also improve uncertainty estimation and robustness by16 making the loss landscape smoother in the same manner as spatial smoothing.17 1 INTRODUCTION18 Azulay & Weiss , 2019 ) .33 Bayesian neural networks ( BNNs ) , such as Monte Carlo34 ( MC ) dropout ( Gal & Ghahramani , 2016 ) , provide a35 probabilistic representation of NN weights . They com-36 bine a number of models selected based on weight prob-37 ability to make predictions of desired results . Thanks to38 this feature , BNNs have been widely used in the areas of uncertainty estimation ( Kendall & Gal , 2017 ) 39 and robustness ( Ovadia et al. , 2019 ) . They are also promising in other fields like out-of-distribution40 detection ( Malinin & Gales , 2018 ) and meta-learning ( Yoon et al. , 2018 ) .41 Nevertheless , there remains a significant challenge that prohibits their use in practice . BNNs require42 an ensemble size of up to fifty to achieve high predictive performance , which results in a fiftyfold43 increase in computational cost ( Kendall & Gal , 2017 ; Loquercio et al. , 2020 ) . Therefore , if BNNs44 can achieve high predictive performance merely with a handful of ensembles , they could be applied45 to a much wider range of areas.46 1.1 PRELIMINARY47 We would first like to discuss BNN inference in detail , then move on to Vector-Quantized BNN48 ( VQ-BNN ) inference ( Park et al. , 2021 ) , an efficient approximated BNN inference.49 BNN inference . Suppose we have access to posterior probability of NN weight p ( w|D ) for training dataset D. The predictive result of BNN is given by the following predictive distribution : p ( y|x0 , D ) = ∫ p ( y|x0 , w ) p ( w|D ) dw ( 1 ) wherex0 is observed input data vector , y is output vector , and p ( y|x , w ) is the probabilistic prediction parameterized by the result of NN for an input x and weight w. In most cases , the integral can not be solved analytically . Thus , we use the MC estimator to approximate it as follows : p ( y|x0 , D ) ' N−1∑ i=0 1 N p ( y|x0 , wi ) ( 2 ) wherewi ∼ p ( w|D ) and N is the number of the samples . The equation indicates that BNN inference50 is ensemble average of NN predictions for one observed data point as shown on the left of Fig . 2.51 Using N neural networks in the ensemble would requires N times more computational complexity52 than one NN execution.53 1.00 Data-complemented BNN inference . To reduce the computational cost of BNN inference , VQBNN ( Park et al. , 2021 ) executes NN for an observed data only once and complements the result with previously calculated predictions for other data . If we have access to previous predictions , the computational performance of VQ-BNN becomes comparable to that of one NN execution . To be specific , VQ-BNN inference is : p ( y|x0 , D ) ' N−1∑ i=0 π ( xi|x0 ) p ( y|xi , wi ) ( 3 ) where π ( xi|x0 ) is the importance of data xi with respect to the observed data x0 , and it is defined as a54 similarity betweenxi andx0 . p ( y|x0 , w0 ) is the newly calculated prediction , and { p ( y|x1 , w1 ) , · · · } 55 are previously calculated predictions . To accurately infer the results , the previous predictions should56 consist of predictions for “ data similar to the observed data ” .57 Thanks to the temporal consistency of real-world data streams , aggregating predictions for similar58 data in data streams is straightforward . Since temporally proximate data sequences tend to be similar,59 we can memorize recent predictions and calculates their average using exponentially decreasing60 importance . In other words , VQ-BNN inference for data streams is simply temporal smoothing of61 recent predictions as shown in the middle of Fig . 2.62 VQ-BNN has two limitations , although it may be a promising approach to obtain reliable results63 in an efficient way . First , it was only applicable to data streams such as video sequences . Applying64 VQ-BNN to images is challenging because it is impossible to memorize all similar images in advance.65 Second , Park et al . ( 2021 ) used VQ-BNN only in the testing phase , not in the training phase . We find66 that ensembling predictions for similar data helps in NN training by smoothing the loss landscape.67 1.2 MAIN CONTRIBUTION68 1 Spatially neighboring points in visual imagery tend to be similar , as do feature maps of convolu-69 tional neural networks ( CNNs ) . By exploiting this spatial consistency , we propose spatial smoothing70 as a method of ensembling nearby feature maps to improve the efficiency of ensemble size in BNN71 inference . The right side of Fig . 2 visualizes spatial smoothing aggregating neighboring feature maps.72 2 We empirically demonstrate that spatial smoothing improves the efficiency in vision tasks , such73 as image classification on CIFAR ( Krizhevsky et al. , 2009 ) and ImageNet ( Russakovsky et al. , 2015 ) ,74 without any additional training parameters . Figure 3 shows that negative log-likelihood ( NLL ) of75 “ MC dropout + spatial smoothing ” with an ensemble size of two is comparable to that of vanilla MC76 dropout with an ensemble size of fifty . We also demonstrate that spatial smoothing improves accuracy,77 uncertainty , and robustness all at the same time . Figure 1 shows that spatial smoothing improves both78 the accuracy and uncertainty of various deterministic and Bayesian NNs with an ensemble size of79 fifty on CIFAR-100.80 3 Global average pooling ( GAP ) ( Lin et al. , 2014 ; Zhou et al. , 2016 ) , pre-activation ( He et al.,81 2016b ) , and ReLU6 ( Krizhevsky & Hinton , 2010 ; Sandler et al. , 2018 ) have been widely used in vision82 tasks . However , their motives are largely justified by the experiments . We provide an explanation for83 these methods by addressing them as special cases of spatial smoothing . Experiments support the84 claim by showing that the methods improve not only accuracy but also uncertainty and robustness.85 2 PROBABILISTIC SPATIAL SMOOTHING86 To improve the computational perfor-87 mance of BNN inference , VQ-BNN88 ( Park et al. , 2021 ) executes NN pre-89 diction only once and complements90 the result with previously calculated91 predictions . The key to the success92 of this approach largely depends on93 the collection of previous predictions94 for proximate data . Gathering tempo-95 rally proximate data and their predic-96 tions from data streams is easy be-97 cause recent data and predictions can98 be aggregated using temporal consis-99 tency . On the other hand , gathering100 time-independent proximate data , e.g . images , is more difficult because they lack such consistency.101 2.1 MODULE ARCHITECTURE FOR ENSEMBLING NEIGHBORING FEATURE MAP POINTS102 So instead of temporal consistency , we use spatial consistency—where neighboring pixels of images103 are similar—for real-world images . Under this hypothesis , we take the feature maps as predictions104 and aggregate neighboring feature maps.105 Most CNN architectures , including ResNet , consist of multiple stages that begin with increasing the number of channels while reducing the spatial dimension of the input volume . We decompose an entire BNN inference into several steps by rewriting each stage in a recurrence relation as follows : p ( zi+1|zi , D ) = ∫ p ( zi+1|zi , wi ) p ( wi|D ) dwi ( 4 ) where zi is input volume of the i-th stage , and the first and the last volume are input data and106 output . wi and p ( wi|D ) are NN weight in the i-th stage and its probability . p ( zi+1|zi , wi ) is output107 probability of zi+1 with respect to the input volume zi . To derive the probability from the output108 feature map , we transform each point of the feature map into a Bernoulli distribution . To do so , a109 composition of tanh and ReLU , a function from value of range [ −∞ , ∞ ] into probability , is added110 after each stage . Put shortly , we use neural networks for point-wise binary feature classification.111 Since Eq . ( 4 ) is a kind of BNN inference , it can be approximated using Eq . ( 3 ) . In other words , each112 stage predicts feature map points only once and complements predictions with similar feature maps.113 Under spatial consistency , it averages probabilities of spatially neighboring feature map points , which114 is well known as blur operation in image processing . For the sake of implementation simplicity,115 average pooling with a kernel size of 2 and a stride of 1 is used as a box blur . This operation ensembles116 four neighboring probabilities with the same importances.117 In summary , as shown in Fig . 4 , we propose the following probabilistic spatial smoothing layer : Smooth ( z ) = Blur ◦ Prob ( z ) ( 5 ) where Prob ( · ) is a point-wise function from a feature map to probability , and Blur ( · ) is importance-118 weighted average for ensembling spatially neighboring probabilities from feature maps . Smooth layer119 is added after each stage . Prob and Blur are further elaborated below.120 Prob : Feature map to probability . Prob is a function that transforms a real-valued feature map into probability . We use tanh–ReLU composition for this purpose . However , tanh is commonly known to suffer from the vanishing gradient problem . To alleviate this issue , we propose the following temperature-scaled tanh : tanhτ ( z ) = τ tanh ( z/τ ) ( 6 ) where τ is a hyperparameter called temperature . τ is 1 in conventional tanh and ∞ in identity121 function . tanhτ imposes an upper bound on a value , but does not limit the upper bound to 1.122 MC dropout MC dropout + Smooth . An unnormalized probability , ranging from 0 to τ , is allowed as the output of Prob . Then , thanks to the linearity of integration , we obtain an unnormalized predictive distribution accordingly . Taking this into account , we propose the following Prob : Prob ( z ) = ReLU ◦ tanhτ ( z ) ( 7 ) where τ > 1 . We empirically determine τ to minimize NLL , a metric that measures both accuracy123 and uncertainty . See Fig . B.3 for more detailed ablation studies . In addition , we expect upper-bounded124 functions , e.g. , ReLU6 ( z ) = ReLU ◦ min ( z , 6 ) and feature map scaling z/τ with τ > 1 which125 is BatchNorm , to be able to replace tanhτ in Prob ; and as expected , these alternatives improve126 uncertainty estimation in addition to accuracy . See Appendix C.2 and Appendix C.3 for detailed127 discussions on activation ( ReLU ◦ BatchNorm ) and ReLU6 as Prob.128 Blur : Averaging neighboring probabilities . Blur averages the probabilities from feature maps . We primarily use the average pool with a kernel size of 2 and a stride of 1 as the implementation of Blur for the sake of simplicity . Nevertheless , we could generalize Blur by using the following depth-wise convolution , which acts on each input channel separately , with non-trainable kernel K = 1 ||k||21 k ⊗ k > ( 8 ) where k is a 1D matrix , e.g. , k ∈ { ( 1 ) , ( 1 , 1 ) , ( 1 , 2 , 1 ) , ( 1 , 4 , 6 , 4 , 1 ) } . Different ks derive different129 importances for neighboring feature maps . We empirically show that most Blurs improve the130 predictive performance and that optimalK varies by model . For more ablation studies , see Table B.2.131 2.2 HOW DOES SPATIAL SMOOTHING HELP OPTIMIZATION ? 132 We present theoretical and empirical aspects to show that spatial smoothing ensembles feature maps.133 Feature map variance . BNNs have two types of uncertainties : One is model uncertainty and the134 other is data uncertainty ( Park et al. , 2021 ) . These randomnesses increase the variance of the feature135 maps . To demonstrate that spatial smoothing is an ensemble , we use the following proposition:136 Proposition 1 . Ensembles reduce the variance of predictions.137 We omit the proof since it is straightforward . In our context , predictions are output feature maps of a138 stage . We investigate model and data uncertainties of the predictions along NN layers to show that139 spatial smoothing reduces the randomnesses and ensembles feature maps . Figure 5 shows the model140 uncertainty and data uncertainty of Bayesian ResNet including MC dropout layers . In this figure , the141 uncertainty of MC dropout ’ s feature map only accumulates , and almost monotonically increases in142 every NN layer . In contrast , the uncertainty of “ MC dropout + spatial smoothing ” ’ s feature map is143 significantly decreases at the end of stages , suggesting that the smoothing layers ensemble the feature144 map . In other words , they make the feature map more accurate and stabilized input volumes for the145 next stages . In addition , consistently , the spatial smoothing layer close to the last layer significantly146 improves performance because it reduces the uncertainty of predictions largely . See Fig . B.5 for more147 detailed results . Deterministic NNs do not have model uncertainty but data uncertainty . Therefore,148 spatial smoothing improves the performance of deterministic NNs as well as Bayesian NNs.149 Fourier analysis . We also analyze spatial smoothing through the lens of Fourier transform:150 Proposition 2 . Ensembles filter high-frequency signals.151 The proof is provided in Eqs . ( 16 ) to ( 17 ) . Figure 6b shows the 2D Fourier transformed output152 feature map at the end of the stage 1 . This figure reveals that MC dropout almost does not affect153 low-frequency ( < 0.3π ) ranges , and it adds high-frequency ( ≥ 0.3π ) noises . Since spatial smoothing154 is a low-pass filter , it effectively filters high-frequency signals , including the noises caused by MC155 dropout.156 We also find that CNNs are particularly vulnerable to high-frequency noises . To demonstrate this157 claim , following Shao et al . ( 2021 ) , we measure accuracy with respect to data with frequency-based158 random noise xnoise = x0 +F−1 ( F ( δ ) Mf ) , where x0 is clean data , F ( · ) and F−1 ( · ) are Fourier159 transform and inverse Fourier transform , δ is random noise , and Mf is frequency mask as shown160 in Fig . 6a . Figure 6c exhibits the results . In sum , high-frequency noises , including those caused by161 MC dropout , significantly impair accuracy . Spatial smoothing improves the robustness by effectively162 removing high-frequency noises.163 Loss landscape . Lastly , we show that the randomness hinders NN training as follows:164 Proposition 3 . Randomness of predictions sharpens the loss landscape , and ensembles flatten it.165 The proof is provided in Eqs . ( 18 ) to ( 25 ) . Since a sharp loss function disturbs NN optimization166 ( Keskar et al. , 2017 ; Santurkar et al. , 2018 ; Foret et al. , 2020 ) , reducing the uncertainty helps NN167 learn strong representations . For example , training phase NN ensemble averages out the randomness,168 and it flattens the loss function . In consequence , an ensemble of BNN outputs in training phase169 significantly improves the predictive performance . See Fig . D.4 for numerical results . However , we170 do not use training phase ensemble because it significantly increases the training time . Instead , we171 use spatial smoothing as a method that ensembles feature maps without sacrificing training time.172 We visualizes the loss landscapes ( Li et al. , 2018 ) , the contours of NLL on training dataset . Figure 8b173 shows that the loss landscapes of MC dropout fluctuate and have irregular surfaces due to the174 randomness . As Li et al . ( 2018 ) ; Foret et al . ( 2020 ) pointed out,175 this may lead to poor generalization and predictive performance.176 Spatial smoothing reduces randomness as discussed above , and177 spatial smoothing aids in optimization by stabilizing and flattening178 the loss landscape of BNN as shown in Fig . 8c.179 Furthermore , we use Hessian to quantitatively represent the sharp-180 ness of the loss landscapes . Figure 7 shows the Hessian max eigen-181 value spectra of the models in Fig . 8 with a batch size of 128 , which182 reveals that spatial smoothing reduces the magnitude of Hessian183 eigenvalues and suppresses outliers . Since large Hessian eigenval-184 ues disturb NN training ( Ghorbani et al. , 2019 ) , we come to the185 same conclusion that spatial smoothing helps NN optimization . See186 Appendix C.1 for a more detailed description of the configurations187 of the Hessian max eigenvalue spectra . In addition , from these188 observations , we propose the conjecture that the flatter the loss189 landscape , the better the uncertainty estimation , and vice versa.190 2.3 REVISITING GLOBAL AVERAGE POOLING191 The success of GAP classifier in image classification is192 indisputable . The initial motivation and the most widely193 accepted explanation for this success is that GAP prevents194 overfitting by using far fewer parameters than multi-layer195 perceptron ( MLP ) ( Lin et al. , 2014 ) . However , we discover196 that the explanation is poorly supported . We compares197 GAP with other classifiers including MLP . Contrary to198 popular belief , Table 1 suggests that MLP does not overfit199 the training dataset . MLP underfits or gives comparable200 performance to GAP on the training dataset . On the test201 dataset , GAP provides better results compared with MLP . See Table C.1 for more detailed results.202 Our argument is that GAP is an extreme case of spatial smoothing . In other words , GAP is successful203 because it ensembles feature maps and smoothens the loss landscape to help optimization . To support204 this claim , we visualizes the loss landscape of MLP as shown in Fig . 8a . It is chaotic compared to205 that of GAP as shown in Fig . 8b . Hessian shows the consistent results as demonstrated by Fig . 7.206 3 EXPERIMENTS207 This section presents two experiments . The first experiment is image classification through which208 we show that spatial smoothing not only improves the ensemble efficiency , but also the accuracy,209 uncertainty , and robustness of both deterministic NN and MC dropout . The second experiment is210 semantic segmentation on data streams through which we show that spatial smoothing and temporal211 smoothing ( Park et al. , 2021 ) are complementary . See Appendix A for more detailed configurations.212 1.00 Three metrics are measured in these experiments : NLL ( ↓1 ) , accuracy ( ↑ ) , and expected calibration213 error ( ECE , ↓ ) ( Guo et al. , 2017 ) . NLL represents both accuracy and uncertainty , and is the most214 widely used as a proper scoring rule . ECE measures discrepancy between accuracy and confidence.215 3.1 IMAGE CLASSIFICATION216 This section mainly discuss ResNet ( He et al. , 2016a ) . Table E.1 also discuss other settings that217 show the same trend : e.g. , VGG ( Simonyan & Zisserman , 2015 ) , ResNeXt ( Xie et al. , 2017 ) ,218 and pre-activation models ( He et al. , 2016a ) . Spatial smoothing also improves deep ensemble219 ( Lakshminarayanan et al. , 2017 ) , another non-Bayesian probabilistic NN method . See Fig . E.1.220 Performance . Fig . 3 and Fig . 9 show the predictive performances of ResNet-18 on CIFAR-100221 and ResNet-50 on ImageNet , respectively . The results indicate that spatial smoothing improves both222 accuracy and uncertainty in many respects . Let us be more specific . First , spatial smoothing improves223 the efficiency of ensemble size . In these examples , the NLL of “ MC dropout + spatial smoothing ” 224 with an ensemble size of 2 is comparable to or even better than that of MC dropout with an ensemble225 size of 50 . In other words , “ MC dropout + spatial smoothing ” is 25× faster than MC dropout with226 a similar predictive performance . Second , the predictive performance of “ MC dropout + spatial227 smoothing ” is better than that of MC dropout , at an ensemble size of 50 . Third , spatial smoothing228 improves the predictive performance of deterministic NN , as well as MC dropout.229 Robustness . To evaluate robustness against data corruption , we230 measure predictive performance of ResNet-18 on CIFAR-100-231 C ( Hendrycks & Dietterich , 2019 ) . This dataset consists of data232 corrupted by 15 different types , each with 5 levels of intensity233 each . We use mean corruption NLL ( mCNLL , ↓ ) , the averages234 of NLL over intensities and corruption types , to summarize the235 performance of corrupted data in a single value . See Eq . ( 32 ) for236 a more rigorous definition . Figure 10 shows that spatial smoothing237 not only improves the efficiency but also corruption robustness238 across a whole range of ensemble size . See Fig . E.3 for more239 details . Spatial smoothing also improves adversarial robustness240 and perturbation consistency ( ↑ ) ( Hendrycks & Dietterich , 2019 ; 241 Zhang , 2019a ) , shift-transformation invariance . See Table E.2,242 Table E.3 , and Fig . E.4 for more details.243 3.2 SEMANTIC SEGMENTATION244 Table 2 summarizes the result of semantic segmentation on CamVid dataset ( Brostow et al. , 2008 ) 245 that consists of real-world 360×480 pixels videos . The table shows that spatial smoothing improves246 predictive performance , which is consistent with the image classification experiment . Moreover , the247 result reveals that spatial smoothing and temporal smoothing ( Park et al. , 2021 ) are complementary.248 See Table E.4 for more results.249 1We use arrows to indicate which direction is better . Spatial smoothing can be compared with prior works in the following areas.251 Anti-aliased CNNs . Local means ( Zhang , 2019a ; Zou et al. , 2020 ; Vasconcelos et al. , 2020 ; Sinha252 et al. , 2020 ) were introduced for the shift-invariance of deterministic CNNs in image classification.253 They were motivated to prevent the aliasing effect of subsampling . Although the local filtering can254 result in a loss of information , Zhang ( 2019a ) experimentally observed an increase in accuracy that255 was beyond expectation . We provide a fundamental explanation for this phenomenon : Local means256 are a spatial ensemble . An ensemble not only improves accuracy , but also uncertainty and robustness257 of deterministic and Bayesian NNs . In Fig . F.1 , we also show that the predictive performance258 improvement is not due to anti-aliasing of local mean . See Appendix F for more discussion on local259 means . For a discussion on non-local means ( Wang et al. , 2018 ) and self-attention ( Dosovitskiy et al.,260 2021 ) , see Section 5.261 Sampling-free BNNs . Sampling-free BNNs ( Hernández-Lobato & Adams , 2015 ; Wang et al.,262 2016 ; Wu et al. , 2019 ) predict results based on a single or couple of NN executions . To this end , it is263 assumed that posterior and feature maps follow Gaussian distributions . However , the discrepancy264 between reality and assumption accumulates in every NN layer . Consequently , to the best of our265 knowledge , most of the sampling-free BNNs could only be applied to shallow models , such as LeNet,266 and were tested on small datasets . Postels et al . ( 2019 ) applied sampling-free BNNs to SegNet ; 267 nonetheless , Park et al . ( 2021 ) argued that they do not predict well-calibrated results.268 Efficient deep ensembles . Deep ensemble ( Lakshminarayanan et al. , 2017 ; Fort et al. , 2019 ) is269 another probabilistic NN approach for predicting reliable results . BatchEnsemble ( Wen et al. , 2020 ; 270 Dusenberry et al. , 2020 ) ensembles over a low-rank subspace to make deep ensemble more efficient.271 Depth uncertainty network ( Antoran et al. , 2020 ) aggregates feature maps from different depths of272 a single NN to predict results efficiently . Despite being robust against data corruption , it provides273 weaker predictive performance compared to deterministic NN and MC dropout.274 5 DISCUSSION275 We propose spatial smoothing , a simple yet efficient module to improve BNN . Three different per-276 spectives , namely , feature map variance , Fourier analysis , and loss landscape , suggest that spatial277 smoothing ensembles feature maps . The limitation of spatial smoothing is that designing its compo-278 nents requires inductive bias . In other words , the optimal shape of the blur kernel is model-dependent.279 We believe this problem can be solved by introducing self-attention ( Vaswani et al. , 2017 ) . Self-280 attentions for computer vision ( Dosovitskiy et al. , 2021 ; Touvron et al. , 2021 ; Carion et al. , 2020 ) 281 can be deemed as trainable importance-weighted ensembles of feature maps . The observation that282 Transformers are more robust than expected ( Bhojanapalli et al. , 2021 ; Shao et al. , 2021 ) supports this283 claim . Therefore , using self-attentions to generalize spatial smoothing would be a promising future284 work because it not only expands our work , but also helps deepen our understanding of self-attention.285 REPRODUCIBILITY STATEMENT286 To ensure reproducibility , we provide comprehensive resources , such as code and experimental details.287 The codebase will be released as open source under the Apache License 2.0 . See the supplemental288 material for the code . Appendix A provides the specifications of all models used in this work . Detailed289 experimental setup including hyperparameters and ablation study are also available in Appendix A290 and Appendix B. De-facto image datasets are used for all experiments as described in Appendix A.291 REFERENCES292 Javier Antoran , James Allingham , and José Miguel Hernández-Lobato . Depth uncertainty in neural293 networks . Advances in Neural Information Processing Systems , 2020.294 Aharon Azulay and Yair Weiss . Why do deep convolutional networks generalize so poorly to small295 image transformations ? Journal of Machine Learning Research , 2019.296 Srinadh Bhojanapalli , Ayan Chakrabarti , Daniel Glasner , Daliang Li , Thomas Unterthiner , and297 Andreas Veit . Understanding robustness of transformers for image classification . In Proceedings298 of the IEEE/CVF International Conference on Computer Vision , 2021.299 Gabriel J Brostow , Jamie Shotton , Julien Fauqueur , and Roberto Cipolla . Segmentation and recog-300 nition using structure from motion point clouds . In European Conference on Computer Vision.301 Springer , 2008.302 Nicolas Carion , Francisco Massa , Gabriel Synnaeve , Nicolas Usunier , Alexander Kirillov , and Sergey303 Zagoruyko . End-to-end object detection with transformers . In European Conference on Computer304 Vision . Springer , 2020.305 Alexey Dosovitskiy , Lucas Beyer , Alexander Kolesnikov , Dirk Weissenborn , Xiaohua Zhai , Thomas306 Unterthiner , Mostafa Dehghani , Matthias Minderer , Georg Heigold , Sylvain Gelly , et al . An image307 is worth 16x16 words : Transformers for image recognition at scale . In International Conference308 on Learning Representations , 2021.309 Michael Dusenberry , Ghassen Jerfel , Yeming Wen , Yian Ma , Jasper Snoek , Katherine Heller , Balaji310 Lakshminarayanan , and Dustin Tran . Efficient and scalable bayesian neural nets with rank-1311 factors . In International Conference on Machine Learning . PMLR , 2020.312 Logan Engstrom , Brandon Tran , Dimitris Tsipras , Ludwig Schmidt , and Aleksander Madry . Exploring313 the landscape of spatial robustness . In International Conference on Machine Learning . PMLR,314 2019.315 Pierre Foret , Ariel Kleiner , Hossein Mobahi , and Behnam Neyshabur . Sharpness-aware minimization316 for efficiently improving generalization . In International Conference on Learning Representations,317 2020.318 Stanislav Fort , Huiyi Hu , and Balaji Lakshminarayanan . Deep ensembles : A loss landscape perspec-319 tive . arXiv preprint arXiv:1912.02757 , 2019.320 Jonathan Frankle , David J Schwab , and Ari S Morcos . Training batchnorm and only batchnorm:321 On the expressive power of random features in cnns . In International Conference on Learning322 Representations , 2021.323 Yarin Gal and Zoubin Ghahramani . Dropout as a bayesian approximation : Representing model324 uncertainty in deep learning . In International Conference on Machine Learning . PMLR , 2016.325 Robert Geirhos , Patricia Rubisch , Claudio Michaelis , Matthias Bethge , Felix A Wichmann , and326 Wieland Brendel . Imagenet-trained cnns are biased towards texture ; increasing shape bias improves327 accuracy and robustness . In International Conference on Learning Representations , 2019.328 Behrooz Ghorbani , Shankar Krishnan , and Ying Xiao . An investigation into neural net optimization329 via hessian eigenvalue density . In International Conference on Machine Learning . PMLR , 2019.330 Ian J Goodfellow , Jonathon Shlens , and Christian Szegedy . Explaining and harnessing adversarial331 examples . International Conference on Learning Representations , 2015.332 Priya Goyal , Piotr Dollár , Ross Girshick , Pieter Noordhuis , Lukasz Wesolowski , Aapo Kyrola,333 Andrew Tulloch , Yangqing Jia , and Kaiming He . Accurate , large minibatch sgd : Training imagenet334 in 1 hour . arXiv preprint arXiv:1706.02677 , 2017.335 Chuan Guo , Geoff Pleiss , Yu Sun , and Kilian Q Weinberger . On calibration of modern neural336 networks . In International Conference on Machine Learning . PMLR , 2017.337 Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Deep residual learning for image338 recognition . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,339 2016a.340 Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Identity mappings in deep residual341 networks . In European Conference on Computer Vision . Springer , 2016b.342 Dan Hendrycks and Thomas Dietterich . Benchmarking neural network robustness to common343 corruptions and perturbations . In International Conference on Learning Representations , 2019.344 José Miguel Hernández-Lobato and Ryan Adams . Probabilistic backpropagation for scalable learning345 of bayesian neural networks . In International Conference on Machine Learning . PMLR , 2015.346 Elad Hoffer , Tal Ben-Nun , Itay Hubara , Niv Giladi , Torsten Hoefler , and Daniel Soudry . Augment347 your batch : Improving generalization through instance repetition . In Proceedings of the IEEE/CVF348 Conference on Computer Vision and Pattern Recognition , 2020.349 A Kendall , V Badrinarayanan , and R Cipolla . Bayesian segnet : Model uncertainty in deep convolu-350 tional encoder-decoder architectures for scene understanding . In BMVC , 2017.351 Alex Kendall and Yarin Gal . What uncertainties do we need in bayesian deep learning for computer352 vision ? Advances in Neural Information Processing Systems , 2017.353 Nitish Shirish Keskar , Dheevatsa Mudigere , Jorge Nocedal , Mikhail Smelyanskiy , and Ping Tak Peter354 Tang . On large-batch training for deep learning : Generalization gap and sharp minima . In355 International Conference on Learning Representations , 2017.356 Alex Krizhevsky and Geoff Hinton . Convolutional deep belief networks on cifar-10 . Unpublished357 manuscript , 2010.358 Alex Krizhevsky , Geoffrey Hinton , et al . Learning multiple layers of features from tiny images . 2009.359 Balaji Lakshminarayanan , Alexander Pritzel , and Charles Blundell . Simple and scalable predictive360 uncertainty estimation using deep ensembles . In Advances in Neural Information Processing361 Systems , 2017.362 Hao Li , Zheng Xu , Gavin Taylor , Christoph Studer , and Tom Goldstein . Visualizing the loss landscape363 of neural nets . In Advances in Neural Information Processing Systems , 2018.364 Min Lin , Qiang Chen , and Shuicheng Yan . Network in network . In International Conference on365 Learning Representations , 2014.366 Antonio Loquercio , Mattia Segu , and Davide Scaramuzza . A general framework for uncertainty367 estimation in deep learning . IEEE Robotics and Automation Letters , 2020.368 Aleksander Madry , Aleksandar Makelov , Ludwig Schmidt , Dimitris Tsipras , and Adrian Vladu.369 Towards deep learning models resistant to adversarial attacks . In International Conference on370 Learning Representations , 2018.371 A Malinin and M Gales . Predictive uncertainty estimation via prior networks . In Advances in Neural372 Information Processing Systems . Curran Associates , Inc. , 2018.373 Yaniv Ovadia , Emily Fertig , Jie Ren , Zachary Nado , David Sculley , Sebastian Nowozin , Joshua374 Dillon , Balaji Lakshminarayanan , and Jasper Snoek . Can you trust your model ’ s uncertainty ? 375 evaluating predictive uncertainty under dataset shift . In Advances in Neural Information Processing376 Systems , 2019.377 Namuk Park , Taekyu Lee , and Songkuk Kim . Vector quantized bayesian neural network inference378 for data streams . In AAAI Conference on Artificial Intelligence , 2021.379 Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor380 Killeen , Zeming Lin , Natalia Gimelshein , Luca Antiga , et al . Pytorch : An imperative style,381 high-performance deep learning library . Advances in Neural Information Processing Systems,382 2019.383 Janis Postels , Francesco Ferroni , Huseyin Coskun , Nassir Navab , and Federico Tombari . Sampling-384 free epistemic uncertainty estimation using approximated variance propagation . In Proceedings of385 the IEEE/CVF International Conference on Computer Vision , 2019.386 Olaf Ronneberger , Philipp Fischer , and Thomas Brox . U-net : Convolutional networks for biomedical387 image segmentation . In International Conference on Medical image computing and computer-388 assisted intervention . Springer , 2015.389 Olga Russakovsky , Jia Deng , Hao Su , Jonathan Krause , Sanjeev Satheesh , Sean Ma , Zhiheng Huang,390 Andrej Karpathy , Aditya Khosla , Michael Bernstein , Alexander C. Berg , and Li Fei-Fei . ImageNet391 Large Scale Visual Recognition Challenge . International Journal of Computer Vision , pp . 211–252,392 2015.393 Mark Sandler , Andrew Howard , Menglong Zhu , Andrey Zhmoginov , and Liang-Chieh Chen . Mo-394 bilenetv2 : Inverted residuals and linear bottlenecks . In Proceedings of the IEEE Conference on395 Computer Vision and Pattern Recognition , 2018.396 Shibani Santurkar , Dimitris Tsipras , Andrew Ilyas , and Aleksander Madry . How does batch normal-397 ization help optimization ? Advances in Neural Information Processing Systems , 2018.398 Rulin Shao , Zhouxing Shi , Jinfeng Yi , Pin-Yu Chen , and Cho-Jui Hsieh . On the adversarial robustness399 of visual transformers . arXiv preprint arXiv:2103.15670 , 2021.400 Karen Simonyan and Andrew Zisserman . Very deep convolutional networks for large-scale image401 recognition . In International Conference on Learning Representations , 2015.402 Samarth Sinha , Animesh Garg , and Hugo Larochelle . Curriculum by smoothing . Advances in Neural403 Information Processing Systems , 2020.404 Hugo Touvron , Matthieu Cord , Matthijs Douze , Francisco Massa , Alexandre Sablayrolles , and Hervé405 Jégou . Training data-efficient image transformers & distillation through attention . In International406 Conference on Machine Learning . PMLR , 2021.407 Cristina Vasconcelos , Hugo Larochelle , Vincent Dumoulin , Nicolas Le Roux , and Ross Goroshin.408 An effective anti-aliasing approach for residual networks . arXiv preprint arXiv:2011.10675 , 2020.409 Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N Gomez , Łukasz410 Kaiser , and Illia Polosukhin . Attention is all you need . In Advances in Neural Information411 Processing Systems , 2017.412 Hao Wang , Xingjian Shi , and Dit-Yan Yeung . Natural-parameter networks : A class of probabilistic413 neural networks . Advances in Neural Information Processing Systems , 2016.414 Xiaolong Wang , Ross Girshick , Abhinav Gupta , and Kaiming He . Non-local neural networks . In415 Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018.416 Yeming Wen , Dustin Tran , and Jimmy Ba . Batchensemble : an alternative approach to efficient417 ensemble and lifelong learning . In International Conference on Learning Representations , 2020.418 Anqi Wu , Sebastian Nowozin , Edward Meeds , Richard E Turner , José Miguel Hernández-Lobato,419 and Alexander L Gaunt . Deterministic variational inference for robust bayesian neural networks.420 In International Conference on Learning Representations , 2019.421 Saining Xie , Ross Girshick , Piotr Dollár , Zhuowen Tu , and Kaiming He . Aggregated residual422 transformations for deep neural networks . In Proceedings of the IEEE Conference on Computer423 Vision and Pattern Recognition , 2017.424 Zhewei Yao , Amir Gholami , Kurt Keutzer , and Michael W Mahoney . Pyhessian : Neural networks425 through the lens of the hessian . In 2020 IEEE International Conference on Big Data ( Big Data ) .426 IEEE , 2020.427 Jaesik Yoon , Taesup Kim , Ousmane Dia , Sungwoong Kim , Yoshua Bengio , and Sungjin Ahn.428 Bayesian model-agnostic meta-learning . In Advances in Neural Information Processing Systems,429 2018.430 Sergey Zagoruyko and Nikos Komodakis . Wide residual networks . In BMVC , 2016.431 Hongyang Zhang , Yaodong Yu , Jiantao Jiao , Eric Xing , Laurent El Ghaoui , and Michael Jordan.432 Theoretically principled trade-off between robustness and accuracy . In International Conference433 on Machine Learning . PMLR , 2019.434 Richard Zhang . Making convolutional networks shift-invariant again . In International Conference on435 Machine Learning . PMLR , 2019a.436 Richard Zhang . Official meta-review of making convolutional networks shift-invariant again , 2019b.437 URL https : //openreview.net/forum ? id=SklVEnR5K7 & noteId=rklZnFS-gN.438 Bolei Zhou , Aditya Khosla , Agata Lapedriza , Aude Oliva , and Antonio Torralba . Learning deep439 features for discriminative localization . In Proceedings of the IEEE Conference on Computer440 Vision and Pattern Recognition , 2016.441 Xueyan Zou , Fanyi Xiao , Zhiding Yu , and Yong Jae Lee . Delving deeper into anti-aliasing in convnets.442 In BMVC , 2020.443

Summary: The motivation of this work is on the computational cost of using BNNs in practice, where applications might require a large number of BNNs in an ensemble formation for achieving good performance. The work in this manuscript aims to reduce the computational cost ensemble.  The manuscript's insight for solving the computational cost is to exploit similarities in spatial neighbouring from images. This is a similar approach used in convolutional neural networks. Experimental results show improved efficiency in image tasks, while also considerably reducing the computational cost when compared to competitor methods.