Revisiting Hilbert-Schmidt Information Bottleneck for Adversarial Robustness
Year: 2022

Abstract:
We investigate the HSIC (Hilbert-Schmidt independence criterion) bottleneck as a regularizer for learning an adversarially robust deep neural network classifier. In addition to the usual cross-entropy loss, we add regularization terms for every intermediate layer to ensure that the latent representations retain useful information for output prediction while reducing redundant information. We show that the HSIC bottleneck enhances robustness to adversarial attacks both theoretically and experimentally. In particular, we prove that the HSIC bottleneck regularizer reduces the sensitivity of the classifier to adversarial examples. Our experiments on multiple benchmark datasets and architectures demonstrate that incorporating an HSIC bottleneck regularizer attains competitive natural accuracy and improves adversarial robustness, both with and without adversarial examples during training. Our code and adversarially robust models are publicly available.2

heading: 1 Introduction
text: Adversarial attacks [8, 17, 18, 3, 5] to deep neural networks (DNNs) have received considerable attention recently. Such attacks are intentionally crafted to change prediction outcomes, e.g, by adding visually imperceptible perturbations to the original, natural examples [25]. Adversarial robustness, i.e., the ability of a trained model to maintain its predictive power under such attacks, is an important property for many safety-critical applications [4, 6, 26]. The most common approach to construct adversarially robust models is via adversarial training [34, 36, 30], i.e., training the model over adversarially constructed samples.
Alemi et al. [1] propose using the so-called Information Bottleneck (IB) [27, 28] to ehnance adversarial robustness. Proposed by Tishby and Zaslavsky [28], the information bottleneck expresses a tradeoff between (a) the mutual information of the input and latent layers vs. (b) the mutual information between latent layers and the output. Alemi et al. show empirically that using IB as a learning objective for DNNs indeed leads to better adversarial robustness. Intuitively, the IB objective increases the entropy between input and latent layers; in turn, this also increases the model’s robustness, as it makes latent layers less sensitive to input perturbations.
Nevertheless, mutual information is notoriously expensive to compute. The Hilbert-Schmidt independence criterion (HSIC) has been used as a tractable, efficient substitute in a variety of machine ∗Equal contribution. 2https://github.com/neu-spiral/HBaR
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
learning tasks [31, 32, 33]. Recently, Ma et al. [16] also exploited this relationship to propose an HSIC bottleneck (HB), as a variant to the more classic (mutual-information based) information bottleneck, though not in the context of adversarial robustness.
We revisit the HSIC bottleneck, studying its adversarial robustness properties. In contrast to both Alemi et al. [1] and Ma et al. [16], we use the HSIC bottleneck as a regularizer in addition to commonly used losses for DNNs (e.g., cross-entropy). Our proposed approach, HSIC-Bottleneck-asRegularizer (HBaR) can be used in conjunction with adversarial examples; even without adversarial training, it is able to improve a classifier’s robustness. It also significantly outperforms previous IB-based methods for robustness, as well as the method proposed by Ma et al.
Overall, we make the following contributions:
1. We apply the HSIC bottleneck as a regularizer for the purpose of adversarial robustness. 2. We provide a theoretical motivation for the constituent terms of the HBaR penalty, proving that
it indeed constrains the output perturbation produced by adversarial attacks. 3. We show that HBaR can be naturally combined with a broad array of state of the art adversarial
training methods, consistently improving their robustness. 4. We empirically show that this phenomenon persists even for weaker methods. In particular,
HBaR can even enhance the adversarial robustness of plain SGD, without access to adversarial examples.
The remainder of this paper is structured as follows. We review related work in Sec. 2. In Sec. 3, we discuss the standard setting of adversarial robustness and HSIC. In Sec. 4, we provide a theoretical justification that HBaR reduces the sensitivity of the classifier to adversarial examples. Sec. 5 includes our experiments; we conclude in Sec. 6.

heading: 2 Related Work
text: Adversarial Attacks. Adversarial attacks often add a constrained perturbation to natural inputs with the goal of maximizing classification loss. Szegedy et al. [25] learn a perturbation via boxconstrained L-BFGS that misleads the classifier but minimally distort the input. FGSM, proposed by Goodfellow et al. [8], is a one step adversarial attack perturbing the input based on the sign of the gradient of the loss. PGD [13, 17] generates adversarial examples through multi-step projected gradient descent optimization. DeepFool [18] is an iterative attack strategy, which perturbs the input towards the direction of the decision boundaries. CW [3] applies a rectifier function regularizer to generate adversarial examples near the original input. AutoAttack (AA) [5] is an ensemble of parameter-free attacks, that also deals with common issues like gradient masking [19] and fixed step sizes [17].
Adversarial Robustness. A common approach to obtaining robust models is adversarial training, i.e., training models over adversarial examples generated via the aforementioned attacks. For example, Madry et al. [17] show that training with adversarial examples generated by PGD achieves good robustness under different attacks. DeepDefense [34] penalizes the norm of adversarial perturbations. TRADES [36] minimizes the difference between the predictions of natural and adversarial
examples to get a smooth decision boundary. MART [30] pays more attention to adversarial examples from misclassified natural examples and adds a KL-divergence term between natural and adversarial samples to the cross-entropy loss. We show that our proposed method HBaR can be combined with several such state-of-the-art defense methods and boost their performance.
Information Bottleneck. The information bottleneck (IB) [27, 28] expresses a tradeoff in latent representations between information useful for output prediction and information retained about the input. IB has been employed to explore the training dynamics in deep learning models [23, 22] as well as a learning objective [1, 2]. Fischer [7] proposes a conditional entropy bottleneck (CEB) based on IB and observes its robust generalization ability empirically. Closer to us, Alemi et al. [1] propose a variational information bottleneck (VIB) for supervised learning. They empirically show that training VIB on natural examples provides good generalization and adversarial robustness. We show that HBaR can be combined with various adversarial defense methods enhancing their robustness, but also outperforms VIB [1] when given access only to natural samples. Moreover, we provide theoretical guarantees on how HBaR bounds the output perturbation induced by adversarial attacks.
Mutual Information vs. HSIC. Mutual information is difficult to compute in practice. To address this, Alemi et al. [1] estimate IB via variational inference. Ma et al. [16] replaced mutual information by the Hilbert Schmidt Independence Criterion (HSIC) and named this the HSIC Bottleneck (HB). Like Ma et al. [16], we utilize HSIC to estimate IB. However, our method is different from Ma et al. [16] in several aspects. First, they use HB to train the neural network stage-wise, layer-bylayer, without backpropagation, while we use HSIC bottleneck as a regularization in addition to cross-entropy and optimize the parameters jointly by backpropagation. Second, they only evaluate the model performance on classification accuracy, while we demonstrate adversarial robustness. Finally, we show that HBaR further enhances robustness to adversarial examples both theoretically and experimentally. Greenfeld et al. [9] use HSIC between the residual of the prediction and the input data as a learning objective for model robustness on covariate distribution shifts. Their focus is on robustness to distribution shifts, whereas our work focuses on robustness to adversarial examples, on which HBaR outperforms their proposed objective.

heading: 3 Background

heading: 3.1 Adversarial Robustness
text: In standard k-ary classification, we are given a dataset D = {(xi, yi)}ni=1, where xi ∈ RdX , yi ∈ {0, 1}k are i.i.d. samples drawn from joint distribution PXY . A learner trains a neural network hθ : RdX → Rk parameterized by weights θ ∈ Rdθ to predict Y from X by minimizing
L(θ) = EXY [`(hθ(X), Y )] ≈ 1
n n∑ i=1 `(hθ(xi), yi), (1)
where ` : Rk × Rk → R is a loss function, e.g., cross-entropy. We aim to find a model hθ that has high prediction accuracy but is also adversarially robust: the model should maintain high prediction accuracy against a constrained adversary, that can perturb input samples in a restricted fashion. Formally, prior to submitting a sample x ∈ RdX to the classifier, an adversary may perturb x by an arbitrary δ ∈ Sr, where Sr ⊆ RdX is the `∞-ball of radius r, i.e.,
Sr = B(0, r) = {δ ∈ RdX : ‖δ‖∞ ≤ r}. (2)
The adversarial robustness [17] of a model hθ is measured by the expected loss attained by such adversarial examples, i.e.,
Lr(θ) = EXY [ max δ∈Sr ` (hθ(X + δ), Y ) ] ≈ 1 n n∑ i=1 max δ∈Sr `(hθ(xi + δ), yi). (3)
An adversarially robust neural network hθ can be obtained via adversarial training, i.e., by minimizing the adversarial robustness loss in (3) empirically over the training set D. In practice, this amounts to training via stochastic gradient descent (SGD) over adversarial examples xi + δ (see, e.g., [17]). In each epoch, δ is generated on a per sample basis via an inner optimization over Sr, e.g., via projected gradient descent (PGD) on −L.

heading: 3.2 Hilbert-Schmidt Independence Criterion (HSIC)
text: The Hilbert-Schmidt Independence Criterion (HSIC) is a statistical dependency measure introduced by Gretton et al. [10]. HSIC is the Hilbert-Schmidt norm of the cross-covariance operator between the distributions in Reproducing Kernel Hilbert Space (RKHS). Similar to Mutual Information (MI), HSIC captures non-linear dependencies between random variables. HSIC(X,Y ) is defined as:
HSIC(X,Y ) = EXYX′Y ′ [kX (X,X ′) kY ′ (Y, Y ′)] + EXX′ [kX (X,X ′)]EY Y ′ [kY (Y, Y ′)] − 2EXY [EX′ [kX (X,X ′)]EY ′ [kY (Y, Y ′)]] ,
(4)
where X ′, Y ′ are independent copies of X , Y , respectively, and kX , kY are kernels.
In practice, we often approximate HSIC empirically. Given n i.i.d. samples {(xi, yi)}ni=1 drawn from PXY , we estimate HSIC via:
ĤSIC(X,Y ) = (n− 1)−2 tr (KXHKYH) , (5)
where KX and KY are kernel matrices with entries KXij = kX(xi, xj) and KYij = kY (yi, yj), respectively, and H = I− 1n11 > is a centering matrix.

heading: 4 Methodology
text: In this section, we present our method, HSIC bottleneck as regularizer (HBaR) as a means to enhance a classifier’s robustness. The effect of HBaR for adversarial robustness is illustrated in Figure 1; the HSIC bottleneck penalty reduces the sensitivity of the classifier to adversarial examples. We provide a theoretical justification for this below, in Theorems 1 and 2, but also validate the efficacy of the HSIC bottleneck extensively with experiments in Section 5.

heading: 4.1 HSIC Bottleneck as Regularizer for Robustness
text: Given a feedforward neural network hθ : RdX → Rk parameterized by θ with M layers, and an input r.v. X , we denote by Zj ∈ RdZj , j ∈ {1, . . . ,M}, the output of the j-th layer under input X (i.e., the j-th latent representation). We define our HBaR learning objective as follows:
L̃(θ) = L(θ) + λx M∑ j=1 HSIC(X,Zj)− λy M∑ j=1 HSIC(Y,Zj), (6)
where L is the standard loss given by Eq. (1) and λx, λy ∈ R+ are balancing hyperparameters. Together, the second and third terms in Eq. (6) form the HSIC bottleneck penalty. As HSIC measures dependence between two random variables, minimizing HSIC(X,Zi) corresponds to removing redundant or noisy information contained in X . Hence, this term also naturally reduces the influence of an adversarial attack, i.e., a perturbation added on the input data. This is intuitive, but we also provide theoretical justification in the next subsection. Meanwhile, maximizing HSIC(Y,Zi) encourages this lack of sensitivity to the input to happen while retaining the discriminative nature of the classifier, captured by dependence to useful information w.r.t. the output label Y . Note that minimizing HSIC(X,Zi) alone would also lead to the loss of useful information, so it is necessary to keep the HSIC(Y,Zi) term to make sure Zi is informative enough of Y .
The overall algorithm is described in Alg. 1. In practice, we perform Stochastic Gradient Descent (SGD) over L̃: both L and HSIC can be evaluated empirically over batches. For the latter, we use the estimator (5), restricted over the current batch. As we have m samples in a mini-batch, the complexity of calculating the empirical HSIC (5) is O(m2dZ̄) [24] for a single layer, where dZ̄ = maxj dZj . Thus, the overall complexity for (6) is O(Mm
2dZ̄). This computation is highly parallelizable, thus, the additional computation time of HBaR is small when compared to training a neural network via cross-entropy only.
Algorithm 1: Robust Learning with HBaR Input: input sample tuples {(xi, yi)}ni=1, kernel function kx, ky, kz , a neural network hθ parameterized by θ, mini-batch size m, learning rate α. Output: parameter of classifier θ while θ has not converged do
Sample a mini-batch of size m from input samples. Forward Propagation: calculate zi and hθ(x). Compute kernel matrices for X , Y and Zi using kx, ky, kz respectively inside mini-batch. Compute L̃(θ) via (6), where HSIC is evaluated empirically via (5). Backward Propagation: θ ← θ − α∇L̃(θ).
end

heading: 4.2 Combining HBaR with Adversarial Examples
text: HBaR can also be naturally applied in combination with adversarial training. For r > 0 the magnitude of the perturbations introduced in adversarial examples, one can optimize the following objective instead of L̃(θ) in Eq. (6):
L̃r(θ) = Lr(θ) + λx M∑ j=1 HSIC(X,Zj)− λy M∑ j=1 HSIC(Y, Zj), (7)
whereLr is the adversarial loss given by Eq. (3). This can be used instead ofL in Alg. 1. Adversarial examples need to be used in the computation of the gradient of the loss Lr in each minibatch; these need to be computed on a per sample basis, e.g., via PGD over Sr, at additional computational cost. Note that the natural samples (xi, yi) in a batch are used to compute the HSIC bottleneck regularizer.
The HBaR penalty can similarly be combined with other adversarial learning methods and/or used with different means for selecting adversarial examples, other than PGD. We illustrate this in Section 5, where we combine HBaR with state-of-the-art adversarial learning methods TRADES [36] and MART [30].

heading: 4.3 HBaR Robustness Guarantees
text: We provide here a formal justification for the use of HBaR to enhance robustness: we prove that regularization terms HSIC(X,Zj), j = 1, . . . ,M lead to classifiers which are less sensitive to input perturbations. For simplicity, we focus on the case where k = 1 (i.e., binary classification). Let Z ∈ RdZ be the latent representation at some arbitrary intermediate layer of the network. That is, Z = Zj , for some j ∈ {1, . . . ,M}; we omit the subscript j to further reduce notation clutter. Then hθ = (g ◦ f), where f : RdX → RdZ maps the inputs to this intermediate layer, and g : RdZ → R maps the intermediate layer to the final layer. Then, Z = f(X) and g(Z) = hθ(X) ∈ R are the latent and final outputs, respectively. Recall that, in HBaR, HSIC(X,Z) is associated with kernels kX , kZ . We make the following technical assumptions:
Assumption 1. Let X ⊆ RdX , Z ⊆ RdZ be the supports of random variables X , Z, respectively. We assume that both hθ and g are continuous and bounded functions in X , Z , respectively, i.e.:
hθ ∈ C(X ), g ∈ C(Z). (8)
Moreover, we assume that all functions hθ and g we consider are uniformly bounded, i.e., there exist 0 < MX ,MZ <∞ such that:
MX = max hθ∈C(X ) ‖hθ‖∞ and MZ = max g∈C(Z) ‖g‖∞. (9)
The continuity stated in Assumption 1 is natural, if all activation functions are continuous. Boundedness follows if, e.g., X , Z are closed and bounded (i.e., compact), or if activation functions are bounded (e.g., softmax, sigmoid, etc.).
Assumption 2. We assume kernels kX , kZ are universal with respect to functions hθ and g that satisfy Assumption 1, i.e., if F and G are the induced RKHSs for kernels kX and kZ , respectively, then for any hθ, g that satisfy Assumption 1 and any ε > 0 there exist functions h′ ∈ F and g′ ∈ G such that ||hθ − h′||∞ ≤ ε and ||g − g′||∞ ≤ ε. Moreover, functions in F and G are uniformly bounded, i.e., there exist 0 < MF ,MG <∞ such that for all h′ ∈ F and all g′ ∈ G:
MF = max f ′∈F ‖f ′‖∞ and MG = max g′∈G ‖g′‖∞. (10)
We note that several kernels used in practice are universal, including, e.g., the Gaussian and Laplace kernels. Moreover, given that functions that satisfy Assumption 1 are uniformly bounded by (9), such kernels can indeed remain universal while satisfying (10) via an appropriate rescaling.
Our first result shows that HSIC(X,Z) at any intermediate layer Z bounds the output variance:
Theorem 1. Under Assumptions 1 and 2, we have:
HSIC(X,Z) ≥ MFMG MXMZ sup θ Var(hθ(X)). (11)
The proof of Theorem 1 is in Appendix B in the supplement. We use a result by Greenfeld and Shalit [9] that links HSIC(X,Z) to the supremum of the covariance of bounded continuous functionals over X and Z . Theorem 1 indicates that the regularizer HSIC(X,Z) at any intermediate layer naturally suppresses the variability of the output, i.e., the classifier prediction hθ(X). To see this, observe that by Chebyshev’s inequality [20] the distribution of hθ(X) concentrates around its mean when Var(hθ(X)) approaches 0. As a result, bounding HSIC(X,Z) inherently also bounds the (global) variability of the classifier (across all parameters θ). This observation motivates us to also maximize HSIC(Y, Z) to recover essential information useful for classification: if we want to achieve good adversarial robustness as well as good predictive accuracy, we have to strike a balance between HSIC(X,Z) and HSIC(Y,Z). This perfectly aligns with the intuition behind the information bottleneck [28] and the well-known accuracy-robustness trade off [17, 36, 29, 21]. We also confirm this experimentally: we observe that both additional terms (the standard loss and HSIC(Y,Z)) are necessary for ensuring good prediction performance in practice (see Table 3).
Most importantly, by further assuming that features are normal, we can show that HSIC bounds the power of an arbitrary adversary, as defined in Eq. (3):
Theorem 2. Assume that X ∼ N (0, σ2I). Then, under Assumptions 1 and 2, we have:3 r √ −2 log o(1)dXMZ σMFMG HSIC(X,Z) + o(r) ≥ E[|hθ(X + δ)− hθ(X)|], for all δ ∈ Sr. (12)
The proof of Theorem 2 can also be found in Appendix C in the supplement. We again use the result by Greenfeld and Shalit [9] along with Stein’s Lemma [15], that relates covariances of Gaussian r.v.s and their functions to expected gradients. In particular, we apply Stein’s Lemma to the bounded functionals considered by Greenfeld and Shalit by using a truncation argument. Theorem 2 implies that HSIC(X,Z) indeed bounds the output perturbation produced by an arbitrary adversary: suppressing HSIC sufficiently can ensure that the adversary cannot alter the out-
put significantly, in expectation. In particular, if HSIC(X,Z) = o (
σMFMG√ −2log o(1)dXMZ
) , then
limr→0 supδ∈Sr E[|hθ(X + δ)− hθ(X)|]/r = 0, i.e., the output is almost constant under small input perturbations.

heading: 5 Experiments

heading: 5.1 Experimental Setting
text: We experiment with three standard datasets, MNIST [14], CIFAR-10 [12] and CIFAR-100 [12]. We use a 4-layer LeNet [17] for MNIST, ResNet-18 [11] and WideResNet-28-10 [35] for CIFAR-10,
3Recall that for functions f, g : R→ R we have f = o(g) if limr→0 f(r)g(r) = 0.
and WideResNet-28-10 [35] for CIFAR-100. We use cross-entropy as loss L(θ). Licensing information for all existing assets can be found in Appendix D in the supplement.
Algorithms. We compare HBaR to the following non-adversarial learning algorithms: CrossEntropy (CE), Stage-Wise HSIC Bottleneck (SWHB) [16], XIC [9], and Variational Information Bottleneck (VIB) [1]. We also incorporate HBaR to several adversarial learning algorithms, as described in Section 4.2, and compare against the original methods, without the HBaR penalty. The adversarial methods we use are: Projected Gradient Descent (PGD) [17], TRADES [36], and MART [30]. Further details and parameters can be found in Appendix E in the supplement.
Performance Metrics. For all methods, we evaluate the obtained model hθ via the following metrics: (a) Natural (i.e., clean test data) accuracy, and adversarial robustness via test accuracy under (b) FGSM, the fast gradient sign attack [8], (c) PGDm, the PGD attack with m steps used for the internal PGD optimization [17], (d) CW, the CW-loss within the PGD framework [3], and (e) AA, AutoAttack [5]. All five metrics are reported in percent (%) accuracy. Following prior literature, we set step size to 0.01 and radius r = 0.3 for MNIST, and step size as 2/255 and r = 8/255 for CIFAR-10 and CIFAR-100. All attacks happen during the test phase and have full access to model parameters (i.e., are white-box attacks). All experiments are carried out on a Tesla V100 GPU with 32 GB memory and 5120 cores.

heading: 5.2 Results
text: Combining HBaR with Adversarial Examples. We show how HBaR can be used to improve robustness when used as a regularizer, as described in Section 4.2, along with state-of-the-art adversarial learning methods. We run each experiment by five times and report the mean natural test accuracy and adversarial robustness of all models on MNIST, CIFAR-10, and CIFAR-100 datasets by four architectures in Table 1 and Table 2. Combined with all adversarial training baselines, HBaR consistently improves adversarial robustness against all types of attacks on all datasets. The resulting improvements are larger than 2 standard deviations (that range between 0.05-0.2) in most cases; we report the results with standard deviations in Appendix G in the supplement. Although natural accuracy is generally restricted by the trade-off between robustness and accuracy [36], we observe that incorporating HBaR comes with an actual improvement over natural accuracy in most cases.
0 20 40 60 80 100 Epoch
0 2 4 6 8
10
Ad ve sa ria
l R ob
us tn es
s (
%
)
MNIST by LeNet: CE only HBaR-high (λx=1, λy=50) HBaR-low (λx=0.001, λy=0.005)
0 20 40 60 80 100 Epoch
0
10
20
30
40
50
HS IC (X , Z
_M )
0 20 40 60 80 100 Epoch
6.5
7.0
7.5
8.0
8.5
9.0
HS IC (Y , Z
_M )
0 20 40 60 80 100 Epoch
93 94 95 96 97 98 99
100
Na tu ra l A
cc ur ac y (%
)
0 20 40 60 80 100 Epoch
0 2 4 6 8
10
Ad ve
sa ria
l R ob
us tn es s (
% )
0 50 100 150 200 250 300 Epoch
0
10
20
30
40
50
Ad ve sa ria
l R ob
us tn es
s ( % )
CIFAR-10 by ResNet-18: CE only HBaR-high (λx=0.006, λy=0.05) HBaR-low (λx=0.001, λy=0.05)
0 50 100 150 200 250 300 Epoch
0
10
20
30
40
50
HS IC (X , Z
_M )
0 50 100 150 200 250 300 Epoch
1 2 3 4 5 6 7 8 9 HS IC (Y , Z _M )
0 50 100 150 200 250 300 Epoch
80 82 84 86 88 90 92 94 96 Na tu ra l A cc ur ac y (% )
0 50 100 150 200 250 300 Epoch
0
10
20
30
40
50
Ad ve
sa ria
l R ob
us tn es s (
% )
(a) HSIC(X,ZM ) (b) HSIC(Y, ZM ) (c) Natural Accuracy (d) Adv. Robustness
Figure 3: Visualization of the HBaR quantities (a) HSIC(X,ZM ), (b) HSIC(X,ZM ), (c) natural test accuracy, and (d) adversarial robustness against PGD attack (PGD40 and PGD20 on MNIST and CIFAR-10, respectively) as a function of training epochs, on MNIST by LeNet (top) and CIFAR-10 by ResNet (bottom). Different colored lines correspond to CE, HBaR-high (HBaR with high weights λ), and HBaR-low (HBaR method small weighs λ). HBaR-low parameters are selected so that the values of the loss L and each of the HSIC terms are close after the first epoch.
Adversarial Robustness Analysis without Adversarial Training. Next, we show that HBaR can achieve modest robustness even without adversarial examples during training. We evaluate the robustness of HBaR on CIFAR-10 by ResNet-18 against various adversarial attacks, and compare HBaR with other information bottleneck penalties without adversarial training in Figure 2. Specifically, we compare the robustness of HBaR with other IB-based methods under various attacks and hyperparameters. Our proposed HBaR achieves the best overall robustness against all three types of attacks while attaining competitive natural test accuracy. Interestingly, HBaR achieves natural accuracy (95.27%) comparable to CE (95.32%) which is much higher than VIB (92.35%), XIC (92.93%) and SWHB (59.18%). We observe SWHB underperforms HBaR on CIFAR-10 for both natural accuracy and robustness. One possible explanation may be that when the model is deep, minimizing HSIC without backpropagation, as in SWHB, does not suffice to transmit the learned information across layers. Compared to SWHB, HBaR backpropagates over the HSIC objective through each intermediate layer and computes gradients only once in each batch, improving accuracy and robustness while reducing computational cost significantly.
Synergy between HSIC Terms. Focusing on ZM , the last latent layer, Figure 3 shows the evolution per epoch of: (a) HSIC(X,ZM ), (b) HSIC(Y,ZM ), (c) natural accuracy (in %), and (d) adversarial robustness (in %) under PGD attack on MNIST and CIFAR-10. Different lines correspond to CE, HBaR-high (HBaR with high weights λ), and HBaR-low (HBaR method small weighs λ). HBaR-low parameters are selected so that the values of the loss L and each of the HSIC terms are close after the first epoch. Figure 3(c) illustrates that all three settings achieve good natural accuracy on both datasets. However, in Figure 3(d), only HBaR-high, that puts sufficient weight on HSIC terms, attains relatively high adversarial robustness. In Figure 3(a), we see that CE leads to high HSIC(X,ZM ) for the shallow LeNet, but low in the (much deeper) ResNet-18, even lower than HBaR-low. Moreover, we also see that the best performer in terms of adversarial robustness, HBaR-high, lies in between the other two w.r.t. HSIC(X,ZM ). Both of these observations indi-
cate the importance of the HSIC(Y,ZM ) penalty: minimizing HSIC(X,ZM ) appropriately leads to good adversarial robustness, but coupling learning to labels via the third term is integral to maintaining useful label-related information in latent layers, thus resulting in good adversarial robustness. Figure 3(b) confirms this, as HBaR-high achieves relatively high HSIC(Y,ZM ) on both datasets.
Figure 4 provides another perspective of the same experiments via the learning dynamics on the HSIC plane. We again observe that the best performer in terms of robustness HBaR-high lies in between the other two methods, crucially attaining a much higher HSIC(Y,ZM ) than HBaR-low. Moreover, for both HBaR methods, we clearly observe the two distinct optimization phases first observed by Shwartz-Ziv and Tishby [23] in the context of the mutual information bottleneck: the fast empirical risk minimization phase, where the neural network tries to learn a meaningful representation by increasing HSIC(Y,ZM ) regardless of information redundancy (HSIC(X,ZM ) increasing), and the representation compression phase, where the neural network turns its focus onto compressing the latent representation by minimizing HSIC(X,ZM ), while maintaining highly label-related information. Interestingly, the HBaR penalty produces the two-phase behavior even though our networks use ReLU activation functions; Shwartz et al. [23] only observed these two optimization phases on neural networks with tanh activation functions, a phenomenon further confirmed by Saxe et al. [22].
Ablation Study. Motivated by the above observations, we turn our attention to how the three terms in the loss function in Eq. (6) affect HBaR. As illustrated in Table 3, removing any part leads to either a significant natural accuracy or robustness degradation. Specifically, using L(θ) only (row [i]) lacks adversarial robustness; removing L(θ) (row [ii]) or the penalty on Y (row [iii]) degrades natural accuracy significantly (a similar result was also observed in [2]); finally, removing the penalty on X improves the natural accuracy while degrading adversarial robustness. The three terms combined together by proper hyperparameters λx and λy (row [v]) achieve both high natural accuracy and adversarial robustness. We provide a comprehensive ablation study on the sensitivity of λx and λy and draw conclusions in Appendix F in the supplement (Tables 7 and 8).

heading: 6 Conclusions
text: We investigate the HSIC bottleneck as regularizer (HBaR) as a means to enhance adversarial robustness. We theoretically prove that HBaR suppresses the sensitivity of the classifier to adversarial examples while retaining its discriminative nature. One limitation of our method is that the robustness gain is modest when training with only natural examples. Moreover, a possible negative societal impact is overconfidence in adversarial robustness: over-confidence in the adversarially-robust models produced by HBaR as well as other defense methods may lead to overlooking their potential failure on newly-invented attack methods; this should be taken into account in safety-critical applications like healthcare [6] or security [26]. We extend the discussion on the limitations and potential negative societal impacts of our work in Appendix H and I, respectively, in the supplement.

heading: 7 Acknowledgements
text: The authors gratefully acknowledge support by the National Science Foundation under grants CCF-1937500 and CNS-2112471, and the National Institutes of Health under grant NHLBI U01HL089856.

Review summaries:
To enhance the adversarial robustness of deep models, this paper proposes to use HSIC bottleneck as a regularizer. Unlike previous work trained with adversarial samples, the proposed learning objective drops redundant information from the input to the latent representations. Theoretical analysis that HSIC regularizer can reduce the output variance is provided.