Towards Adversarial Robustness of Bayesian Neural Network through Hierarchical Variational Inference

Recent works have applied Bayesian Neural Network ( BNN ) to adversarial training , and shown the improvement of adversarial robustness via the BNN ’ s strength of stochastic gradient defense . However , we have found that in general , the BNN loses its stochasticity after its training with the BNN ’ s posterior . As a result , the lack of the stochasticity leads to weak regularization effect to the BNN , which increases KL divergence in ELBO from variational inference . In this paper , we propose an enhanced Bayesian regularizer through hierarchical variational inference in order to boost adversarial robustness against gradient-based attack . Furthermore , we also prove that the proposed method allows the BNN ’ s stochasticity to be elevated with the reduced KL divergence . Exhaustive experiment results demonstrate the effectiveness of the proposed method by showing the improvement of adversarial robustness , compared with adversarial training ( Madry et al. , 2018 ) and adversarial-BNN ( Liu et al. , 2019 ) under PGD attack and EOT-PGD attack to the L∞ perturbation on CIFAR-10/100 , STL-10 , and Tiny-ImageNet . 1 INTRODUCTION . Deep neural networks have achieved impressive performance in a wide variety of machine learning tasks . Despite the breakthrough outcomes , deep neural networks are easily deceived from adversarial attack with the carefully crafted perturbations ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ; Chen et al. , 2017 ; Carlini & Wagner , 2017 ; Papernot et al. , 2017 ; Eykholt et al. , 2018 ; Madry et al. , 2018 ) . Injecting these perturbations into clean inputs ( i.e. , adversarial examples ) , which are imperceptible to the human eyes , fools the estimators in the deep neural networks . Weak reliability due to the invisible perturbations has affected security problems in deep learning applications ( Apruzzese et al. , 2019 ; Wang et al. , 2019b ; Sagduyu et al. , 2019 ; Rosenberg et al. , 2020 ) . To defend such adversarial examples , many algorithms have been studied to improve adversarial robustness so far . Adversarial training , where deep neural networks are trained on adversarial examples , is one of the few defense strategies against strong adversarial attacks ( Huang et al. , 2015 ; Zantedeschi et al. , 2017 ; Kurakin et al. , 2017 ; Madry et al. , 2018 ; Athalye et al. , 2018a ; Liu et al. , 2018 ) . Among them , Madry et al . ( 2018 ) has shown that adversarially trained networks can be robust to white-box attacks with the knowledge of the network parameters . Besides , most of the above studies have agreed with that adversarial training shows an effective adversarial robustness against several white-box attacks . Meanwhile , adversarial training and BNN have been combined to improve adversarial robustness with stochastic approach through variational inference . In fact , the variational inference maximizes Evidence Lower Bound ( ELBO ) to find an approximate posterior closely following the true posterior for the machine learning tasks ( Graves , 2011 ; Kingma & Welling , 2014 ; Blundell et al. , 2015 ; Hernández-Lobato & Adams , 2015 ) . Based on the variational inference , adversarial training with BNN has accomplished the achievement of adversarial robustness by implicitly using the approximate posterior against the adversarial perturbations ( Mescheder et al. , 2017 ; Ye & Zhu , 2018 ) . They have focused on training the network parameters or spaces itself on the maximum ELBO without obtaining the approximate posterior directly . Contrary to the above studies , Liu et al . ( 2019 ) presents an adversarial training with BNN , called “ adversarial-BNN ” to deal with the approximate posterior explicitly . They straightforwardly learn Gaussian parameters ( e.g. , mean and variance ) of the approximate posterior as follows : w ∼ N ( µ , σ2 ) , instead of the weight parameters . Alternatively , the weight parameters are sampled by the approximate posterior , such that w = µ + σ , where the stochastic sampler is gererated from ∼ N ( 0 , 1 ) . The stochastic sampler provides the change of the weight parameters with the learned Gaussian parameters . The variation of them creates stochastic gradient , helping the improvement of adversarial robustness ( Carbone et al. , 2020 ) . However , we find that in general , the approximate posterior ’ s variance converges to zero-like small value as follows : w ∼ N ( µ , σ2 ≈ 0 ) after training the BNN with its posterior . Although the stochastic sampler helps the weight parameters to change , they become fixed-like parameters , such that w = µ+σ ( ≈ 0 ) . The lack of their stochasticity causes the BNN ’ s stochasticity to be vanished so that the BNN can not easily respond to slightly different inputs within the same class . In other words , the vanished stochasticity breaks the regularization effect in the BNN , which increases KL divergence in the ELBO . The broken BNN regularizer produces an ill-posed posterior , thus resulting in weak adversarial robustness . This is because the broken regularizer hinders the maximum ELBO from approximating the true posterior against the adversarial perturbations . Therefore , an enhanced BNN regularizer is required to better approximate the true posterior for adversarial robustness . In this paper , we present the enhanced Bayesian regularizer through hierarchical variational inference in order to boost adversarial robustness compared to the BNN regularizer from variational inference . Furthermore , we also prove that the proposed method significantly intensifies the BNN ’ s stochasticity by introducing a closed form approximation of conjugate prior for the true posterior . In the end , we validate the effectiveness of the proposed method by showing the improvement of adversarial robustness , compared with adversarial training ( Madry et al. , 2018 ) and adversarial-BNN ( Liu et al. , 2019 ) under PGD attack as well as EOT-PGD attack to theL∞ perturbation on CIFAR-10/100 , STL-10 , and Tiny-ImageNet . Our contributions of this paper can be summarized into two-fold as follows . • We newly design an enhanced Bayesian regularizer through hierarchical variational inference built with a concept of the conjugate prior , and verify that the proposed method further strengthens the BNN ’ s stochasticity , compared to the BNN regularizer based on variational inference . • We conduct exhaustive experiments to validate the effectiveness of the proposed method by adversarial robustness , and exhibit the outstanding performance compared with adversarial training and adversarial-BNN under both PGD attack and EOT-PGD attack on four benchmark datasets : CIFAR-10/100 , STL-10 , and Tiny-ImageNet . 2 PRELIMINARY . In this section , we specify the notations used in our paper at first and summarize the related works on adversarial attack/defense , and adversarial training with BNN . Notations . Let x denote the clean image from a given dataset , and y denote the class label corresponding to the clean image . Let D and Dadv indicate each clean dataset and adversarial dataset , such that ( x , y ) ∼ D and ( xadv , y ) ∼ Dadv . A deep neural network f parameterized by weight parameters w is denoted by fw ( x ) . Adversarial examples are represented by xadv = x + δ , where δ denotes the adversarial perturbations . In order to align the experiments in the previous works , we use the cross-entropy loss J ( fw ( x ) , y ) for image classification . Moreover , we regard δ as the L∞ perturbation within γ-ball , such that ‖δ‖∞ ≤ γ . Here , ‖·‖∞ describes the L∞ . 2.1 ADVERSARIAL ATTACK/DEFENSE . Adversarial Attacks . The goal of adversarial attacks is generating the adversarial examples to deceive the prediction of the deep neural networks . Most of them produce the adversarial examples by the gradient of the loss function over the input . Goodfellow et al . ( 2015 ) introduces a single-step attack called Fast Gradient Sign Method ( FGSM ) . Kurakin et al . ( 2017 ) proposes iterative-FGSM with multiple-step attack . Further , Carlini & Wagner ( 2017 ) presents C & W attack to overcome defensive distillation ( Papernot et al. , 2016 ) , encompassing a range of attacks cast from the same optimization framework . On the other hand , Athalye et al . ( 2018a ) analyzes the effectiveness of Projected Gradient Descent ( PGD ) method ( Madry et al. , 2018 ) to perform adversarial attack to the L∞ perturbation . PGD attack is an iterative algorithm by computing the gradient of the loss in the direction of the highest loss and projecting it back to the L∞ perturbation around the clean image . We use the PGD attack for the experiments of adversarial robustness , which can be written as : xadvt+1 = ∏ x , γ [ xadvt + η · sign ( ∇xJ ( fw ( x ) , y ) ∣∣ x=xadvt ) ] , ( 1 ) where ∏ x , γ is the function to project its argument to the surface of x ’ s γ-neighbor ball { x ∣∣ ∥∥xadv − x∥∥∞ ≤ γ } , and η denotes step size . For the adversarial defense utilizing randomized classifiers , an adaptive attack such as Expectation over Transformation ( EOT ) is known to be effective because it allows an attacker to compute the actual gradient over the expected transformation to the stochastic classifiers ( Athalye et al. , 2018a ; b ) . Inspired from the EOT attack , we can modify Eq . ( 1 ) and adjust more efficient PGD attack ( Zimmermann , 2019 ) . xadvt+1 = ∏ x , γ [ xadvt + η · sign ( E w [ ∇xJ ( fw ( x ) , y ) ∣∣ x=xadvt ] ) ] , ( 2 ) Zimmermann ( 2019 ) applies the averaged-gradient over the multiple weight parameters to generate the adversarial examples , and shows that this efficient PGD attack degrades the BNN ’ s adversarial robustness . This is because the averaged-gradient can attack all of the possibly sampled weight parameters in the BNN . In other words , the averaged-gradient attack weakens the BNN ’ s advantage of stochastic gradient defense . In this paper , we call the efficient PGD attack , namely EOT-PGD attack , and take it to validate the effectiveness of the proposed method . Adversarial Defense . The aim of adversarial defense is to secure deep learning networks against adversarial attacks . There are generally three main categories of defense strategies ( Silva & Najafirad , 2020 ; Hao-Chen et al. , 2020 ) . ( 1 ) Gradient masking/obfuscation : a defender intentionally hides the gradient information of deep neural networks in order to confuse adversaries , because most attack algorithms are based on the classifier ’ s gradient information ( Papernot et al. , 2016 ; Buckman et al. , 2018 ; Guo et al. , 2018 ; Dhillon et al. , 2018 ) . ( 2 ) Adversarial detections : they focus on distinguishing whether the input is benign or adversarial ( Metzen et al. , 2017 ; Grosse et al. , 2017 ; Xu et al. , 2017 ) . When they succeed to detect the adversarial attack in the input , the classifiers stop making a decision . ( 3 ) Robust optimization : it is a well-known paradigm that aims to obtain solutions under bounded feasible regions . Especially , its main interest from an adversarial perspective , is improving the classifier ’ s robustness by changing the learning scheme of deep neural networks ( Cisse et al. , 2017 ; Hein & Andriushchenko , 2017 ; Raghunathan et al. , 2018 ; Wong & Kolter , 2018 ) . Adversarial training ( Goodfellow et al. , 2015 ; Kurakin et al. , 2017 ; Madry et al. , 2018 ) is one of the famous robust optimization methods in deep learning fields , which allows the deep neural network to learn robust parameters against the adversarial examples . Madry et al . ( 2018 ) tries to find the optimal weight parameters against adversarial dataset Dadv to improve adversarial robustness under PGD attack to the L∞ perturbation , by minimizing the pre-defined loss L for the machine learning tasks . The formulation can be written as : w∗ = arg min w E ( xadv , y ) ∼Dadv [ L ( fw ( x adv ) , y ) ] , ( 3 ) where xadv is generated from the PGD attack as described in Eq . ( 1 ) . The optimal weight parameters w∗ get the capacity of adversarial defense to confront the adversarial perturbations .

Summary: In this paper the authors study the adversarial robustness of BNNs on large scale datasets. BNNs have been shown to be a more robust learning paradigm due to their uncertainty/stochasticity. Given the empirical observation that adversarially trained BNN posterior variances converge to zero (which the authors need to do much more to show as this is not a well-established phenomena), the authors propose a hierarchical prior where they put a prior over the parameters of the Gaussian prior normally used  in mean-field variational inference. The authors show that performing approximate inference with a hierarchical prior leads to an increased variational posterior variance, which the authors claim is correlated to the observation of increased adversarial robustness.