Revisiting Hilbert-Schmidt Information Bottleneck for Adversarial Robustness

1 Introduction . Adversarial attacks [ 8 , 17 , 18 , 3 , 5 ] to deep neural networks ( DNNs ) have received considerable attention recently . Such attacks are intentionally crafted to change prediction outcomes , e.g , by adding visually imperceptible perturbations to the original , natural examples [ 25 ] . Adversarial robustness , i.e. , the ability of a trained model to maintain its predictive power under such attacks , is an important property for many safety-critical applications [ 4 , 6 , 26 ] . The most common approach to construct adversarially robust models is via adversarial training [ 34 , 36 , 30 ] , i.e. , training the model over adversarially constructed samples . Alemi et al . [ 1 ] propose using the so-called Information Bottleneck ( IB ) [ 27 , 28 ] to ehnance adversarial robustness . Proposed by Tishby and Zaslavsky [ 28 ] , the information bottleneck expresses a tradeoff between ( a ) the mutual information of the input and latent layers vs. ( b ) the mutual information between latent layers and the output . Alemi et al . show empirically that using IB as a learning objective for DNNs indeed leads to better adversarial robustness . Intuitively , the IB objective increases the entropy between input and latent layers ; in turn , this also increases the model ’ s robustness , as it makes latent layers less sensitive to input perturbations . Nevertheless , mutual information is notoriously expensive to compute . The Hilbert-Schmidt independence criterion ( HSIC ) has been used as a tractable , efficient substitute in a variety of machine ∗Equal contribution . 2https : //github.com/neu-spiral/HBaR 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . learning tasks [ 31 , 32 , 33 ] . Recently , Ma et al . [ 16 ] also exploited this relationship to propose an HSIC bottleneck ( HB ) , as a variant to the more classic ( mutual-information based ) information bottleneck , though not in the context of adversarial robustness . We revisit the HSIC bottleneck , studying its adversarial robustness properties . In contrast to both Alemi et al . [ 1 ] and Ma et al . [ 16 ] , we use the HSIC bottleneck as a regularizer in addition to commonly used losses for DNNs ( e.g. , cross-entropy ) . Our proposed approach , HSIC-Bottleneck-asRegularizer ( HBaR ) can be used in conjunction with adversarial examples ; even without adversarial training , it is able to improve a classifier ’ s robustness . It also significantly outperforms previous IB-based methods for robustness , as well as the method proposed by Ma et al . Overall , we make the following contributions : 1 . We apply the HSIC bottleneck as a regularizer for the purpose of adversarial robustness . 2 . We provide a theoretical motivation for the constituent terms of the HBaR penalty , proving that it indeed constrains the output perturbation produced by adversarial attacks . 3 . We show that HBaR can be naturally combined with a broad array of state of the art adversarial training methods , consistently improving their robustness . 4 . We empirically show that this phenomenon persists even for weaker methods . In particular , HBaR can even enhance the adversarial robustness of plain SGD , without access to adversarial examples . The remainder of this paper is structured as follows . We review related work in Sec . 2 . In Sec . 3 , we discuss the standard setting of adversarial robustness and HSIC . In Sec . 4 , we provide a theoretical justification that HBaR reduces the sensitivity of the classifier to adversarial examples . Sec . 5 includes our experiments ; we conclude in Sec . 6 . 2 Related Work . Adversarial Attacks . Adversarial attacks often add a constrained perturbation to natural inputs with the goal of maximizing classification loss . Szegedy et al . [ 25 ] learn a perturbation via boxconstrained L-BFGS that misleads the classifier but minimally distort the input . FGSM , proposed by Goodfellow et al . [ 8 ] , is a one step adversarial attack perturbing the input based on the sign of the gradient of the loss . PGD [ 13 , 17 ] generates adversarial examples through multi-step projected gradient descent optimization . DeepFool [ 18 ] is an iterative attack strategy , which perturbs the input towards the direction of the decision boundaries . CW [ 3 ] applies a rectifier function regularizer to generate adversarial examples near the original input . AutoAttack ( AA ) [ 5 ] is an ensemble of parameter-free attacks , that also deals with common issues like gradient masking [ 19 ] and fixed step sizes [ 17 ] . Adversarial Robustness . A common approach to obtaining robust models is adversarial training , i.e. , training models over adversarial examples generated via the aforementioned attacks . For example , Madry et al . [ 17 ] show that training with adversarial examples generated by PGD achieves good robustness under different attacks . DeepDefense [ 34 ] penalizes the norm of adversarial perturbations . TRADES [ 36 ] minimizes the difference between the predictions of natural and adversarial examples to get a smooth decision boundary . MART [ 30 ] pays more attention to adversarial examples from misclassified natural examples and adds a KL-divergence term between natural and adversarial samples to the cross-entropy loss . We show that our proposed method HBaR can be combined with several such state-of-the-art defense methods and boost their performance . Information Bottleneck . The information bottleneck ( IB ) [ 27 , 28 ] expresses a tradeoff in latent representations between information useful for output prediction and information retained about the input . IB has been employed to explore the training dynamics in deep learning models [ 23 , 22 ] as well as a learning objective [ 1 , 2 ] . Fischer [ 7 ] proposes a conditional entropy bottleneck ( CEB ) based on IB and observes its robust generalization ability empirically . Closer to us , Alemi et al . [ 1 ] propose a variational information bottleneck ( VIB ) for supervised learning . They empirically show that training VIB on natural examples provides good generalization and adversarial robustness . We show that HBaR can be combined with various adversarial defense methods enhancing their robustness , but also outperforms VIB [ 1 ] when given access only to natural samples . Moreover , we provide theoretical guarantees on how HBaR bounds the output perturbation induced by adversarial attacks . Mutual Information vs. HSIC . Mutual information is difficult to compute in practice . To address this , Alemi et al . [ 1 ] estimate IB via variational inference . Ma et al . [ 16 ] replaced mutual information by the Hilbert Schmidt Independence Criterion ( HSIC ) and named this the HSIC Bottleneck ( HB ) . Like Ma et al . [ 16 ] , we utilize HSIC to estimate IB . However , our method is different from Ma et al . [ 16 ] in several aspects . First , they use HB to train the neural network stage-wise , layer-bylayer , without backpropagation , while we use HSIC bottleneck as a regularization in addition to cross-entropy and optimize the parameters jointly by backpropagation . Second , they only evaluate the model performance on classification accuracy , while we demonstrate adversarial robustness . Finally , we show that HBaR further enhances robustness to adversarial examples both theoretically and experimentally . Greenfeld et al . [ 9 ] use HSIC between the residual of the prediction and the input data as a learning objective for model robustness on covariate distribution shifts . Their focus is on robustness to distribution shifts , whereas our work focuses on robustness to adversarial examples , on which HBaR outperforms their proposed objective . 3 Background . 3.1 Adversarial Robustness . In standard k-ary classification , we are given a dataset D = { ( xi , yi ) } ni=1 , where xi ∈ RdX , yi ∈ { 0 , 1 } k are i.i.d . samples drawn from joint distribution PXY . A learner trains a neural network hθ : RdX → Rk parameterized by weights θ ∈ Rdθ to predict Y from X by minimizing L ( θ ) = EXY [ ` ( hθ ( X ) , Y ) ] ≈ 1 n n∑ i=1 ` ( hθ ( xi ) , yi ) , ( 1 ) where ` : Rk × Rk → R is a loss function , e.g. , cross-entropy . We aim to find a model hθ that has high prediction accuracy but is also adversarially robust : the model should maintain high prediction accuracy against a constrained adversary , that can perturb input samples in a restricted fashion . Formally , prior to submitting a sample x ∈ RdX to the classifier , an adversary may perturb x by an arbitrary δ ∈ Sr , where Sr ⊆ RdX is the ` ∞-ball of radius r , i.e. , Sr = B ( 0 , r ) = { δ ∈ RdX : ‖δ‖∞ ≤ r } . ( 2 ) The adversarial robustness [ 17 ] of a model hθ is measured by the expected loss attained by such adversarial examples , i.e. , Lr ( θ ) = EXY [ max δ∈Sr ` ( hθ ( X + δ ) , Y ) ] ≈ 1 n n∑ i=1 max δ∈Sr ` ( hθ ( xi + δ ) , yi ) . ( 3 ) An adversarially robust neural network hθ can be obtained via adversarial training , i.e. , by minimizing the adversarial robustness loss in ( 3 ) empirically over the training set D. In practice , this amounts to training via stochastic gradient descent ( SGD ) over adversarial examples xi + δ ( see , e.g. , [ 17 ] ) . In each epoch , δ is generated on a per sample basis via an inner optimization over Sr , e.g. , via projected gradient descent ( PGD ) on −L . 3.2 Hilbert-Schmidt Independence Criterion ( HSIC ) . The Hilbert-Schmidt Independence Criterion ( HSIC ) is a statistical dependency measure introduced by Gretton et al . [ 10 ] . HSIC is the Hilbert-Schmidt norm of the cross-covariance operator between the distributions in Reproducing Kernel Hilbert Space ( RKHS ) . Similar to Mutual Information ( MI ) , HSIC captures non-linear dependencies between random variables . HSIC ( X , Y ) is defined as : HSIC ( X , Y ) = EXYX′Y ′ [ kX ( X , X ′ ) kY ′ ( Y , Y ′ ) ] + EXX′ [ kX ( X , X ′ ) ] EY Y ′ [ kY ( Y , Y ′ ) ] − 2EXY [ EX′ [ kX ( X , X ′ ) ] EY ′ [ kY ( Y , Y ′ ) ] ] , ( 4 ) where X ′ , Y ′ are independent copies of X , Y , respectively , and kX , kY are kernels . In practice , we often approximate HSIC empirically . Given n i.i.d . samples { ( xi , yi ) } ni=1 drawn from PXY , we estimate HSIC via : ĤSIC ( X , Y ) = ( n− 1 ) −2 tr ( KXHKYH ) , ( 5 ) where KX and KY are kernel matrices with entries KXij = kX ( xi , xj ) and KYij = kY ( yi , yj ) , respectively , and H = I− 1n11 > is a centering matrix .

Summary: To enhance the adversarial robustness of deep models, this paper proposes to use HSIC bottleneck as a regularizer. Unlike previous work trained with adversarial samples, the proposed learning objective drops redundant information from the input to the latent representations. Theoretical analysis that HSIC regularizer can reduce the output variance is provided.