Revisiting Hilbert-Schmidt Information Bottleneck for Adversarial Robustness
Authors: Zifeng Wang
Tong Jian
Aria Masoomi
Stratis Ioannidis
Jennifer Dy
Venue: NeurIPS 2021 Poster (2021)
Keywords: Adversarial Robustness
Hilbert-Schmidt Independence Criterion
Information Bottleneck
Deep Learning

Abstract: We investigate the HSIC (Hilbert-Schmidt independence criterion) bottleneck as a regularizer for learning an adversarially robust deep neural network classifier. In addition to the usual cross-entropy loss, we add regularization terms for every intermediate layer to ensure that the latent representations retain useful information for output prediction while reducing redundant information. We show that the HSIC bottleneck enhances robustness to adversarial attacks both theoretically and experimentally. In particular, we prove that the HSIC bottleneck regularizer reduces the sensitivity of the classifier to adversarial examples. Our experiments on multiple benchmark datasets and architectures demonstrate that incorporating an HSIC bottleneck regularizer attains competitive natural accuracy and improves adversarial robustness, both with and without adversarial examples during training. Our code and adversarially robust models are publicly available.

Revisiting Hilbert-Schmidt Information Bottleneck for Adversarial Robustness
Zifeng Wang"
Tong Jian" Northeastern University jian jian@ece.o. nnuu. edu
Northeastern University zif sifeewaanggeee neu, edu
Aria Masoomi Northeastern University masoomi www.ommiseccortthsetteeee. edu
Stratis Ioannidis Northeastern University ioannidis@ece. neu. edu
Jennifer Dy Northeastern University jdyžece neu edu
a
Abstract
We investigate the HSIC (Hilbert-Schmidt independence criterion) bottleneck as a regularizer for learning an adversarially robust deep neural network classifier. In addition the usual cross-entropy loss, we add regularization terms for every intermediate layer to ensure that the latent representations retain useful informa- tion for output prediction while reducing redundant information. We show that the HSIC bottleneck enhances robustness to adversarial attacks both theoretically and experimentally.
particular, we prove that the HSIC bottleneck regularizer reduces the sensitivity of the classifier to adversarial examples. Our experiments on multiple benchmark datasets and architectures demonstrate that incorporating HSIC bottleneck regularizer attains competitive natural accuracy and improves adversarial robustness, both with and without adversarial examples during train- ing. Our code and adversarially robust models are publicly available
Introduction
3 5 to deep neural networks (DNNs) have received considerable attention recently. Such attacks are intentionally crafted to change prediction outcomes, e.g, by adding visually imperceptible perturbations to the original, natural examples [25]. Adversarial ro- bustness, i.c., the ability of a trained model to maintain its predictive power under such attacks, is an important property for many safety-critical applications [4] 6 26] The most common approach to
Adversarial attacks
construct adversarially robust models is via adversarial training over adversarially constructed samples.
36
30 i.e., training the model
Alemi et al. I propose using the so-called Information Bottleneck (IB) 127 28 to ehnance ad- versarial robustness. Proposed by Tishby and Zaslavsky (28), the information bottleneck expresses tradeoff between (a) the mutual information of the input and latent layers vs. (b) the mutual in- formation between latent layers and the output. Alemi al. show empirically that using as learning objective for DNNs indeed leads to better adversarial robustness. Intuitively, the IB objec- tive increases the entropy between input and latent layers; in turn, this also increases the model's robustness, as it makes latent layers less sensitive to input perturbations.
Nevertheless, mutual information is notoriously expensive to compute. The pendence criterion (HSIC) has been used as a tractable, efficient substitute in variety of machine
inde-
Equal contribution.
htttps://gichub.ccm//eeee www.:/ooooeeucrreeeeeee comm/neu-ggiral/Baall
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

input space
neural network h(-)
output space
decision boundary
hf 8) w/o HBaR
h + 6) with HBaR
class
class 2
Figure 1: Illustration of HBaR for adversarial robustness. neural network trained with HBaR gives a more constrained prediction w.r.t. perturbed inputs. Thus, it less sensitive to adversarial examples.
Recently, Ma et al. [16 also exploited this relationship to propose HSIC bottleneck (HB), as a variant to the more classic nnutual-information based) information bot- tleneck, though not in the context of adversarial robustness.
learning tasks 33 32 33
We revisit the HSIC bottleneck, studying its adversarial robustness properties. contrast to both Alemi al. [ and Ma et al. [16], we use the HSIC bottleneck regularizer in addition to com- monly used losses for DNNs (e.g., cross-entropy). Our proposed approach, Regularizer (HBaR) can be used in conjunction with adversarial examples; even without adversarial training, able to improve a classifier's s robustness. It also significantly outperforms previous IB-based methods for robustness, well the method proposed by Ma et al.
Overall, we make the following contributions:
1. We apply the HSIC bottleneck as regularizer for the purpose of adversarial robustness.
2. We provide a theoretical motivation for the constituent terms of the HBaR penalty, proving that it indeed constrains the output perturbation produced by adversarial attacks.
3. We show that HBaR can be naturally combined with broad array of state of the art adversarial training methods, consistently improving their robustness.
4. We empirically show that this phenomenon persists even for weaker methods. In particular, HBaR can even enhance the adversarial robustness of plain SGD, without access to adversarial examples.
The remainder of this paper is structured as follows. We review related work in Sec. 2 In Sec. diseuss the standard setting of adversarial robustness and HSIC. In Sec. we provide theoret- ical justification that HBaR reduces the sensitivity of the classifier to adversarial examples. Sec. includes experiments; conclude in Sec.
2 Related Work
Adversarial Attacks. Adversarial attacks often add a constrained perturbation to natural inputs with the goal of maximizing classification loss. Szegedy et (25] learn a perturbation via box- constrained L-BFGS that misleads the classifier but minimally distort the input. FGSM, proposed by Goodfellow et [ a one step adversarial attack perturbing input based on sign of the gradient of the loss. PGD [13 [7] generates adversarial examples through multi-step projected gradient descent optimization. DeepFool [18 is an iterative attack strategy, which perturbs the input towards the direction of the decision boundaries. CW B applies rectifier function regularizer to generate adversarial examples near the original input. AutoAttack (AA) (5) is an ensemble of parameter-free attacks, that also deals with common issues like gradient masking 19] and fixed step sizes [17]
Adversarial Robustness. A common approach to obtaining robust models is adversarial training, i.e. training models over adversarial examples generated via the aforementioned attacks. For ex- ample, Madry et al. [17 show that training with adversarial examples generated by PGD achieves good robustness under different attacks. DeepDefense 34 penalizes the norm of adversarial pertur- bations. TRADES [36] minimizes the difference between the predictions of natural and adversarial

---

examples to get a smooth decision boundary. MART [30] pays more attention to adversarial ex- amples from misclassified natural examples and adds KL-divergence term between natural and adversarial samples to the cross-entropy loss. We show that our proposed method HBaR can be combined with several such state-of-the -ort----rtttatt--ee defense methods and boost their performance.
Information Bottleneck. The information bottleneck (IB) 127 28 expresses a tradcoff' in latent representations between information useful for output prediction and information retained about the input. IB has been employed to explore the training dynamics in deep learning models [23 22 as well as learning objective 2 Fischer 7 proposes a conditional entropy bottleneck (CEB) based on IB and observes its robust generalization ability empirically. Closer to us, Alemi et al. [ propose variational information bottleneck (VIB) for supervised learning. They empirically show that training VIB on natural examples provides good generalization and adversarial robustness. We show that HBaR can be combined with various adversarial defense methods enhancing their robustness, but also outperforms VIB [1 when given access only to natural samples. Moreover, we provide theoretical guarantees on how HBaR bounds the output perturbation induced by adversarial attacks.
Mutual Information vs. HSIC. Mutual information is difficult to compute in practice. To address this, Alemi al. M estimate IB via variational inference. Ma al. [T6] replaced mutual information by the Hilbert Schmidt Independence Criterion (HSIC) and named this the HSIC Boutleneck (HB). Like Ma et al. [T),, we utilize HSIC to estimate IB. However, our method is different from Ma et al. [16] in several aspects. First, they use HB to train the neural network stage-wise layer-by- layer, without backpropagation, while we use HSIC bottleneck as regularization in addition to cross-entropy and optimize the parameters jointly by backpropagation. Second, they only evaluate the model performance on classification accuracy, while we demonstrate adversarial robustness. Finally, we show that HBaR further enhances robustness to adversarial examples both theoretically and experimentally. Greenfeld et al. 9 use HSIC between the residual of the prediction and the input data as a learning objective for model robustness on covariate distribution shifts. Their focus is on robustness to distribution shifts, whereas our work focuses on robustness to adversarial examples, on which HBaR outperforms their proposed objective.
3 Background
3.1 Adversarial Robustness
In standard k-ary classification,
given a dataset D {(12, (4) w)] where x; E R We {0, 1)*h are i.i.d. samples drawn from joint distribution Pxy. A learner trains a neural network he : Rdx + Re parameterized by weights 0 e Ied to predict Y from X by minimizing
[()) = Exy
T Σ ((h((z)). v.).
(1)
where e: RR X R + R is loss function, e.g., cross-entropy. We aim to find a model he that has high prediction accuracy but is also adversarially robust: the model should maintain high prediction accuracy against a constrained adversary, that can perturb input samples in a restricted fashion. Formally, prior to submitting sample x E Rdx to the classifier, adversary may perturb by an arbitrary 5 E S,, where S, C Rdx is the for -ball of radius r, i.e.,
S, = = r) = {6 E Rd : =100l> r..
(2)
The adversarial robustness adversarial examples, i.e.,
of a model hi is measured by the expected loss attained by such
(0) Exy
max ll (ho + +)) SES.
max ((ho + v.).
(3)
n
dES.
An adversarially robust neural network he can be obtained via adversarial training, i.e., by mini- mizing the adversarial robustness loss in (3) empirically over the training set D. In practice, this amounts to training via stochastic gradient descent (SGD) over adversarial examples x; + d (see, e.g., [17). In each epoch, is generated on a per sample basis via an inner optimization over S,, e.g., via projected gradient descent (PGD) on L.
3

---

3.2 Hilbert-Schmidt Independence Criterion (HSIC)
The Independence Criterion (HSIC) is statistical dependency measure introduced by Gretton al. [T]] HSIC is the SilbeettSSchiit norm of the cross-covariance operator between the distributions in Reproducing Kernel Hilbert Space (RKHS). Similar to Mutual Information (MI), HSIC captures non-linear dependencies between random variables. HSIC(X, Y) defined as:
HSIC(X, Y) Exyxy [kxx (X, ky (Y, Y
Exxx' [xx (X, X() Byy [ky (Y, YY)
(4)
2Exy [x [kx (X, (X)) By [ky (X,Y")I,,
where X', Y' are independent copies of Y. respectively, and lx, by are kernels.
In practice, we often approximate HSIC empirically. Given n iid. samples {(z,, W)] drawn from Pxy, estimate HSIC via:
HSIC - (KX
(5)
where Kx and Ky are kernel matrices with entries Kx,, = (x(( and Ky, = by by Us). respectively, and H = 111 centering matrix.
4 Methodology
In this section, present method, HSIC bottleneck regularizer (HBaR) means enhance a classifier's 's robustness. The effect of HBaR for adversarial robustness is illustrated in Figure [ the HSIC bottleneck penalty reduces the sensitivity of the classifier to adversarial examples. We provide a theoretical justification for this below, in Theorems and 2 but also validate the efficacy of the HSIC bottleneck extensively with experiments in Section
4.1 HSIC Rottleneck as Regularizer for Robustness
Given a feedforward neural network he Fdx - R² parameterized by 0 with M layers, and an input r.v. we denote by Z, E R E (1..... . M). the output of the j-th layer under input X (i.e., the j-th latent representation), We define our HBaR learning objective follows:
(() = C(0) A Σ HSIC(X, 2₁) X Σ HSIC(Y, 2z),
(6)
where L is the standard loss given by Eq. and de Ay E R4 are balancing hyperparameters.
Together, the second and third terms in Eq. 6 form the HSIC bottleneck penalty. As HSIC measures dependence between two random variables. minimizing HSIC(X, Z.) corresponds removing re- dundant or noisy information contained in X. Hence, this term also naturally reduces the influence of an adversarial attack, i.e., a perturbation added on the input data. This is intuitive, but we also provide theoretical justification in the next subsection. Meanwhile, maximizing HSIC(Y, Z,) en- courages this lack of sensitivity to the input to happen while retaining the discriminative nature of the classifier, captured by dependence to useful information w.r.t. the output label Y. Note that minimizing XSIC Z,) alone would also lead to the loss of useful information, so it is necessary to keep the HSIC(Y, Z,) term to make sure Z, is informative enough of Y.
The overall algorithm is described in Alg. I In practice, we perform Stochastic Gradient Descent (SGD) over L: both L and HSIC can be evaluated empirically over batches. For the latter, we use the estimator (5). restricted over the current batch. As we have m samples in mini-batch, the complexity
calculating the empirical HSIC is O(m²dz) (24 for single layer, where dz max; dz,. Thus, the overall complexity for (6 is O(Mmddz This computation is highly parallelizable, thus, the additional computation time HBaR is small when compared to training neural network via cross-entropy only.

---

Algorithm 1: Robust Learning with HBaR
Input:
input sample tuples {(1,, Mi) )==)) kernel function kz, ky, kiz, a neural network ho parameterized by 0, mini-batch size m, learning rate o.
Output: parameter of classifier 0
while 0 has not converged do
Sample mini-batch of size from input samples.
Forward Propagation: calculate 2; and (z))).
Compute kernel matrices for X, Y and Z, using kas Ky, k2 respectively inside mini-batch.
Compute ((0) via (6), where HSTC evaluated empirically via (5).
Backward Propagation: < 0 - a V(()..
end
4.2
Combining HBaR with Adversarial Examples
HBaR can also be naturally applied in combination with adversarial training. For r >> 0 the magni- tude of the perturbations introduced in adversarial examples, one can optimize the following objec- tive instead of [(0) in Eq. (
- + Am HSIC(X, Xy HSIC(Y, 2j),
where C, is the adversarial loss given by Eq. (3) This can be used instead of in Alg. Adversarial examples need be used in the computation of the gradient of the loss L, in each minibatch; these need to be computed on per sample basis, e.g., via PGD over S,, at additional computational cost. Note that the natural samples (Iri, y/) in batch are used compute the HSIC bottleneck regularizer.
The HBaR penalty can similarly be combined with other adversarial learning methods and/or used with different means for selecting adversarial examples, other than PGD. We illustrate this in Section 5 where we combine HBaR with state-of-the-ar adversarial learning methods TRADES [36] and MART 30
4.3 HBaR Robustness Guarantees
We provide here a formal justification for the use of HBaR to enhance robustness: we prove that regularization terms HSIC(X, Zy). 1,...,M. MM lead classifiers which are less sensitive to input perturbations. For simplicity, we focus on the case where k = (i.e., binary classification). Let Z E dk be the latent representation at some arbitrary intermediate layer of the network. That is, Z = Zj. for some j E (1,..., , M); we omit the subscript to further reduce notation clutter. Then hg = (g o f), where f : Rdx + IRd2 maps the inputs to this intermediate layer, and Rd - TR maps the intermediate layer to the final layer. Then, Z f(X) and g(Z) he (X) E latent and final outputs, respectively. Recall that, in HBaR, HSIC(X, Z) is associated with kernels kx. kz. We make the following technical assumptions:
the
Assumption 1. Let X C Rdx Z C Rdda x be the supports of random variables X, Z, respectively. We assume that both hg and are continuous and bounded functions in X. Z, respectively, i.e.:
hg C(x), E C(Z).
Moreover, we assume that all functions by and g we consider uniformly bounded, i.e. there exist 0 < Mx, Mz < 0C such that:
Mx max hcEC(2))
[Toolloo and Mz max l|gloo. gEC(Z)
The continuity stated in Assumption 1 is natural, if all activation functions are continuous. Bound- edness follows if, e.g., X, 2 are closed and bounded (i.e., compact), or if activation functions are bounded (e.g., softmax, sigmoid, etc.).
5

---

Assumption 2. We assume kernels kx. kz are universal with respect to functions ha and that satisfy Assumption [ i.e. if and 9 are the induced RKHSs for kernels kx and kz. respectively, then for any he, g that satisfy Assumption 7 and any > O there exist functions h' E F and g E such that |[hg 1s S & and ||g - g" gllla S E. Moreover, functions in F and 9 are uniformly bounded, iee, there exist 0 < Ms. Mg < oc such that for all h' E F and all g' E g:
Mf max 1l and Mg - 1lg' grEg
(10)
M
We note that several kernels used in practice are universal, including, e.g., the Gaussian and Laplace kernels. Moreover, given that functions that satisfy Assumption [ are uniformly bounded by ( such kernels can indeed remain universal while satisfying (TÕ) via an appropriate rescaling.
Our first result shows that HSIC(X, 2) at any intermediate layer Z bounds the oulput variance:
Theorem 1. Under Assumptions 7 2 we huve:
My Mx
HSIC(X, 2) Z
sup Var(ho(X)). 8
(1I)
The proof of Theorem 1 is in Appendix B in the supplement. We use a result by Greenfeld and Shalit [9] that links HSIC(X, Z) to the supremum of the covariance of bounded continuous func- tionals over X' and 2. Theorem 1 indicates that the regularizer HSIC(X, 2) at any intermediate layer naturally suppresses the variability of the output, i.e., the elassifier prediction (X). To this, observe that by Chebyshev's inequality (20) the distribution of hg X) concentrates around its mean when Var(he(X)) approaches 0. As result, bounding HSTC(X, X, 7) inherently also bounds the (global) variability of the classifier (across all parameters 0). This observation motivates us to also maximize HSIC((Y Z) to recover essential information useful for classification: if we want to achieve good adversarial robustness as well as good predictive accuracy, we have to strike a balance between HSIC [X Z) and HSIC(Y, Z). This perfectly aligns with the intuition behind the informa- tion bottleneck (28) and the well-known accuracy accuracy-robustness robustness trade off [17] [1]] We also con- firm this experimentally: we observe that both additional terms (the standard loss and HSIC Z)) are necessary for ensuring good prediction performance in practice (see Table 3
Most importantly, by further assuming that features normal, we can show that HSIC bounds the power of an arbitrary adversary, as defined in Eq. (3:
Theorem 2. Assume that X ~ ((0,021) Then, under Assumptions 7 and 2 we have
TV- 2 log M2 HSIC 2) + o(r) 2 (X + +d) - (X)I,, for all 8 E S,.. (12)
a MM
The proof of Theorem 2 can also be found in Appendix C in the supplement. We again use the result by Greenfeld and Shalit (9] along with Stein's Lemma [15]. that relates covariances of Gaussian r.v.s and their functions to expected gradients. In particular, we apply Stein's Lemma to the bounded functionals considered by Greenfeld and Shalit by using a truncation argument. Theorem 2 implies that HSIC(X, Z) indeed bounds the output perturbation produced by an arbi- trary adversary: suppressing HSIC sufficiently can ensure that the adversary cannot alter the out-
put significantly, in expectation. In particular, if HSIC(X, = ( 2log [(1]x M2
My
then
g Eilhg (X + 5) - ho(X)ll/r O, i.e., the output is almost constant under small input perturbations.
5 Experiments
5.1 Experimental Setting
We experiment with three standard datasets, MNIST [4,, CIFAR- 12 and CIFAR-100 [2]. We use 4-layer LeNet [17 for MNIST, ResNet-18 and 22
335 for CIFAR-
Recall that for functions f.g: IR IR we have f o(g) if lim
0.
s(r)

---

Table I: Natural test accuracy (in %), adversarial robustness ((in %) FGSM, PGD, CW, and AA attacked examples) on MNIST and CIFAR-100 of [row i, lii, v] adversarial learning baselines and [row ii, iv, vi] combining HBaR with each correspondingly. Each result the average five runs.
MNIST by LeNet
CHFAR-100 by WideResNet-28-10
Methods
Natural
FGSM
PGD
PGD
CW
AA
Natural
FGSM PGD 10 PGD
CW
AA
PGD
98.40
93.44
94.56
89.63
91.20
86.62
59.91
29.85
26.05
25.38
22.28
20.91
HBaR + PGD
98.66
96.02
96.44
94.35
95.10
91.57
63.84
31.59
27.90
27.21
23.23
21.61
TRADES
97.64
94.73
95.05
93.27
89.66 93.05
60.29
34.19
31.32
30,96
28.20
26.91
HBaR TRADES
97.64
95.23
95.17
93.49
93.47
89.99
60.55
34.57
31.96
31.57
28.72
27.46
MART
98.29
95.57
95.23
93.55
93.45
88.36
58.42
32.94
29.17
28.19
27.31
25.09
HBaR + MART
98.23
96.09
96.08
94.64
94.62
89.99
58,93
33,49
30.72
30.16
28.89
25.21
Table 2: Natural test accuracy (in %), adversarial robustness ((in %) on FGSM, PGD, CW, and AA attacked test examples) on CIFAR- 10 by ResNet- 18 and WideResNNet-2--Ue [row i, iii, v] adversarial learning baselines and [row ii, iv, vi] combining HBaR with each correspondingly. Each result is the average of five runs.
CIFAR- 10 by ResNer-18
CIFAR-IO 10 by WideResNet-28-
Mcthods
Natural
FGSM
PGD
PGD
CW
AA
Natural FGSM PGD
PGD
CW
AA
PGD
84.71
55.95
49.37
47.54
41.17
43.42
86.63
58.53
52.21
50.59
49.32
47.25
HBaR + PGD
85,73
57.13
49.63
48.32
41.80
44.46
87.91
59.69
52.72
51.17
49,52
47.60
TRADES
84.07
58.63
53,21
52.36
50.07
49.38
85.66
61.55
56.62
55.67
54.02
52.71
HBaR TRADES
84.10
58,97
53.76
52.92
51.00
49.43
85.61
62,20
57.30
56,51
54.89
53.53
MART
82.15
59.85
$4.75
53.67
12
47.97
85.94
59.39
51.30
49.46
47.94
45.48
HBaR + MART
82.44
59.86
54.84
53.89
50.53
48.21
85.52
60.54
53.42
51.81
49.32
46.99
and WideResNet-28.. 10
35 for CIFAR-100. We use cross-entropy as loss L()). Licensing infor- mation for all existing assets can be found in Appendix Dj in the supplement.
Algorithms. We compare HBaR to the following non-adversarial learning algorithms: Cross- Entropy (CE), Stage-Wise IISIC Bottleneck (SWIIB) [6]. XIC [9] and Variational Information Bot- rleneck ( (VIB) M. We also incorporate HBaR to several adversarial learning algorithms, as described in Section 4.2 and compare against the original methods, without the HBaR penalty. The adversar- ial methods we use are: Projected Gradient Descent (PGD) [17]. TRADES (36), and MART (30]
Further details and parameters can be found in Appendix E in the supplement.
Performance Metrics. For all methods, evaluate the obtained model ho via the following met- rics: (a) Natural (i.e., clean test data) accuracy, and adversarial robustness via test accuracy under (b) FGSM, the fast gradient sign attack (8. (c) PGD"" the PGD attack with m steps used for the internal PGD optimization [17. (d) CW, the CW-loss within the PGD framework [3], and (e) AA, AutoAAttck [5] All five metrics are reported in percent (%) accuracy. Following prior literature, we set step size to 0.01 and radius r = 0.3 for MNIST, and step size as 2/255 and r 8/255 for CIFAR-10 and CIFAR- 100. All attacks happen during the test phase and have full access to model parameters (i.c., are white-box attacks), All experiments are carried out on Tesla V100 GPU with 32 GB memory and 5120 cores.
5.2 Results
Combining HBaR with Adversarial Examples. We show how HBaR can be used to improve robustness when used as regularizer, as described in Section 4.2 along with state-of-the-ar ad- versarial learning methods. We run each experiment by five times and report the mean natural test accuracy and adversarial robustness of all models on MNIST, CIFAR-10, and CIFAR- 100 datasets by four architectures in Table[ Table Combined with all adversarial training baselines, HBaR consistently improves adversarial robustness against all types of attacks on all datasets. The result- ing improvements are larger than standard deviations (that range between 0.05-0.2) most cases; we report the results with standard deviations in Appendix G in the supplement. Although natural accuracy is generally restricted by the trade-off between robustness and accuracy [36), we observe that incorporating HBaR comes with an actual improvement over natural accuracy in most cases.
7

---

PSG stadks
(a) PGD attacks
(b) CW attack
(c) AA using different radius
Figure 2: CIFAR-10 10 by ResNet-18: Adversarial robustness of IB-based baselines and proposed HBaR under (a) PGD attacks, (h) CW attack by various of constant and (c) AA using different radius. Interestingly, while achieving the highest adversarial robustness under almost all cases, HBaR achieves natural accuracy (95.27%) comparable to CE (95.32%) which much higher than VIB (92.35%), XIC (92.93%) SWHB (59.18%).
MNIST by LeNet:
CE only
HBaR-high (4, 150
HBaR-low =00000,,, A,
CIFAR-10 ResNet-18:
CE only
HBaR-high (1,00,006 0.006. A, 0,05,
HBaR-low (A, (A,-0.001, A, 00.05)
28
303
355
209
350
369
51
330
300
221
30
30
330
2200
350
259
(a) HSIC(X, Zm)
(b) HSIC(Y, 2M) (c) Natural Accuracy
(d) Adv. Robustness
Figure 3: Visualization of the HBaR quantities (a) HSIC(X, Zw). (b) HSIC(X, Zar). (c) natural test accuracy, and (d) adversarial robustness against PGD attack (PGD and PGD MNIST and CIFAR-IO, respectively) function of training epochs, on MNIST by LeNet (top) and CIFAR-10 10 by ResNet (bottom). Different colored lines correspond to CE, HBaR-high high (HBaR with high weights A), and HBaR-low (HBaR method small weighs A). HBaR-low parameters are selected so that the values of the loss L and cach of the HSIC terms are close after the first epoch.
Adversarial Robustness Analysis without Adversarial Training. Next, we show that HBaR can achieve modest robustness even without adversarial examples during training. We evaluate the bustness of HBaR on CIFAR-I0 by ResNet-18 against various adversarial attacks, and compare HBaR with other information bottleneck penalties without adversarial training in Figure 2 Specif- ically, we compare the robustness of HBaR with other IB-based methods under various attacks and hyperparameters. Our proposed HBaR achieves the best overall robustness against all three types of attacks while attaining competitive natural test accuracy. Interestingly, HBaR achieves natural accu- racy (95.2) (9%,77% comparable CE (95.32% which much higher than VIB (92.35%) XIC (92.93%) and SWHB (59.18% We observe SWHB underperforms HBaR on CIFAR- 10 for both natural ac- curacy and robustness. One possible explanation may be that when the model is deep. minimizing HSIC without backpropagation, as in SWHB, does not suffice to transmit the learned information across layers. Compared to SWHB, HBaR backpropagates over the HSIC objective through each intermediate layer and computes gradients only once in each batch, improving accuracy and robust- ness while reducing computational cost significantly.
Synergy between HSIC Terms. Focusing on ZM- the last latent layer, Figure g shows the evo- lution per epoch of: (a) HSIC(X, ZM). (b) HSIC(Y, ZM), (c) natural accuracy (in %), and (d) adversarial robustness (in %) under PGD attack on MNIST and CIFAR-10. Different lines corre- spond to CE, HBaR-high high (HBaR with high weights A), and HBaR-low (HBaR method small weighs A). HBaR-low parameters are selected so that the values of the loss L and each of the HSIC terms are close after the first epoch. Figure (i() illustrates that all three settings achieve good natural ac- curacy on both datasets. However, in Figure d). only HBaR-high. that puts sufficient weight on HSIC terms, attains relatively high adversarial robustness. In Figure Aa), we see that CE leads to high HSIC(X, ZM) for the shallow LeNet, but low in the (much deeper) ResNet-18. 18, even lower than HBaR-low. Moreover, we also see that the best performer in terms of adversarial robustness, HBaR- high, lies in between the other two w.r.t HSIC(X, ZM). Both of these observations indi-
8

---

9.0
10
8.5
88.0
7.5
CE only
tighAttighh,,11,1 1,=50
6.5
epoch
HBaR-low (A,-0 001, 000000)
6.0
10
30
40
50
0
HSICOX, Zu)
(a) MNIST by LeNet
35
30
25
opoch
20
CE only
HBaR-high(AA, A.-0.05)
HBaR: (A,00 001,
50
HSICIX, Zu)
(b) CIFAR-10 by ResNet-18
Figure 4: HSIC plane dynamics versus adversarial robustness. The x-axis plots HSIC between the last inter- mediate layer Zas and the input X, while the y-axis plots HSIC between Zaz and the output Y. The color scale indicates adversarial robustness against PGD attack (PGD and PGD 20 on MNIST and CIFAR-10, re- spectively). The arrows indicate dynamic direction w.r.t. training epochs. Each marker in the figures represents different setting: dots, stars, and triangles represent CE-only, HBaR-high, and HBaR--ow, respectively, compatible with the definition in Figure
cate the importance of the HSIC(Y, ZAr) penalty: minimizing HSIC(X, ZMr) appropriately leads to good adversarial robustness, but coupling learning labels via the third term integral to maintain- ing useful label-related information in latent layers, thus resulting in good adversarial robustness.
Figure 3 confirms this, High achieves relatively high HSIC(Y, aaz) on both datasets.
Figure provides another perspective of the same experiments via the learning dynamics on the HSIC plane. We again observe that the best performer in terms of robustness HBaR-high lies in between the other two methods, crucially attaining a much higher HSTC(Y, Zvr) than HBaR-low. Moreover, for both HBaR methods, we clearly observe the two distinct optimization phases first ob- served by Shwartz-Zij and Tishby [23] in the context of the mutual information bottleneck: the fast empirical risk minimizaation phase, where the neural network tries to learn a meaningful representa- tion by increasing HSIC((Y ZMz) regardless of information redundancy (HSIC(XX Zxz) increasing), and the representation compression phase, where the neural network turns focus onto compress- ing the latent representation by minimizing HSIC(X, ZM) while maintaining highly rebaeed information. Interestingly, the HBaR penalty produces the two-phase behavior even though our networks use ReLU activation functions; Shwartz et al. 2 only observed these two optimization phases on neural networks with tanh activation functions, phenomenon further confirmed by Saxe et al. [2
Ablation Study. Motivated by the above observations, we turn our attention to how the three terms in the loss function in Eq. (6) affect HBaR. As illustrated in Table 3 removing any part leads to either significant natural accuracy or robustness degradation. Specifically, using ((0) only (row (i]) lacks adversarial robustness; removing L(#) (row [ii] or the penalty on Y (row [iii] degrades natural accuracy significantly (a similar result was also observed (2]); finally, removing the penalty on X improves the natural accuracy while degrading adversarial robustness. The three terms combined together by proper hyperparameters X, and X, (row [v]) achieve both high natural accuracy and adversarial robustness. We provide comprehensive ablation study on the sensitivity of XS and X, and draw conclusions in Appendix F in the supplement (Tables 7 and g)

---

Table 3: Ablation study HBaR. Rows [i-iv] indicate the effect of removing each component of the learning objective defined in 66 (row [v]) We evaluate each objective over HSTC(X, ZMr), HSTC(Y, ZMr). natural accuracy (in %). and adversarial robustness %) against PGD PGD on MNIST and CIFAR-I0 respectively. We set A as 1 and 0.006, xy as 50 and 0.05, for MNIST and CIFAR-I0 respectively.
MNIST by LeNet
CIFAR-IO by ResNet-18
Rows
Objectives
HSIC
Natural PGD
HSIC
Natural PGDy
(X, ZMr) (Y, ZM)
X. Zar) (Y, Zxr)
D(M)
99.23
95.32
[i]
Ao D HSIC(X, 2,) A E, HSIC(V.
16.45
8.65
30.08
9.47
44.37
8.72
19.30
8.58
[
((0) + HSICIX, 2ạ) ((0) R. E, HISICYY.))
0.00
0.00
11.38
10.00
0.00
0.00
10.03
10.10
[iv]
56.38
9.00
99.33
43.71
95.50
[v]
((9) A. E, IISICIX, 22) A,E, IISICIY,
15.68
8.89
98.90
8.33
6.07
8,30
95.35
34.85
Conclusions
We investigate the HSIC bottleneck regularizer (HBaR) as means enhance adversarial ro- bustmess. We theoretically prove that HBaR suppresses the sensitivity of the classifier to adversarial examples while retaining its discriminative nature. One limitation of method is that the robust- ness gain is modest when training with only natural examples. Moreover, possible negative soci- etal impact is overconfidence in adversarial robustness: over-confidence in the adversarially-robbn models produced by HBaR as well as other defense methods may lead to overlooking their potential failure newly-invented attack methods; this should be taken into account in safety-critical appli- cations like healthcare [6] or security (26). We extend the discussion on the limitations and potential negative societal impacts of our work Appendix H and respectively, the supplement.
7
Acknowledgements
The authors gratefully acknowledge support by the National Science Foundation under grants CCF-1937500 and CNS-2112471, and the National Institutes of Health under grant NHLBI U01HL089856.
10

---

References
[1] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational infor- mation bottleneck. In ICLR, 2017.
[2] Rana Ali Amjad and Bernhard C Geiger. How (not) train your neural network using the information bottleneck principle. arXiv preprint /XXxv:/802.097000 2018.
[3] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy, pages 39-57, 2017.
[4] Alesia Cheriikova, Alina Oprea, Cristina Nita-Rotaru and BaekGyu Kim. Are self-driving cars secure? evasion attacks against deep neural networks for steering angle prediction. In IEEE Security and Privacy Workshops, pages 132-137, 2019.
[5] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, volume 119, pages 2206-22262 2216, 2020.
[6] Samuel G Finlayson, John D Bowers, Joichi Ito, Jonathan I. Zittrain, Andrew I. Beam, and Isaac S Kohane. Adversarial attacks on medical machine learning. Science, 1636633)
1289, 2019.
[7] Ian Fischer. The conditional entropy bottleneck. Entropy, 22(9))999, 2020.
[8] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. In ICLR, 2015.
[9] Daniel Greenfeld and Uri Shalit. Robust learning with the hilbert-schmidt independence crite- rion. In /CML, pages 3759-3768. PMLR, 2020.
[10] Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In International Conference on Algorithmic Learning Theory, pages 63-77. Springer, 2005.
[11] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[12] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[13] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning scale. In ICLR, 2017.
[14] Y. Lecun, L.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 96(11))2278-2324 1998.
[15] Jun Liu. Siegel's formula via stein's identities. Statistics & Probability Letters, 21(3):247- 251, 1994.
[16] Wan-Duo Kurt Ma. JP Lewis, and W Bastiaan Kleijn. The hsic bottleneck: Deep learning without pacc-propagation. In AAAI, volume 34, pages 5085-5092, 2020.
[17] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
[18] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In CVPR, pages 2574-2582, 2016.
[19] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Anan- thram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pages 506-519, 2017.
[20] Athanasios Papoulis and Saunders. Probability, random variables and stochastic processes. 1989.

---

[21] Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John Duchi, and Percy Liang. Un- derstanding and mitigating the tradeoff between robustness and accuracy. arXiv preprint arXiv:2002. 16716, 2020.
[22] Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment. 2019(12):124020) 2019.
[23] Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprini
2017.
[24] Le Song, Alex Smola, Arthur Gretton, Justin Bedo, and Karsten Borgwardt. Feature selection via dependence maximization. Journal of Machine Learning Research, 13(5), 2012.
[25] Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Good- fellow, and Rob Fergus. Intriguing properties of neural networks. In /CIR, 2014.
[26] Simen Thys, Wiebe Van Ranst, and Toon Goedemé. Fooling automated surveillance cameras: adversarial patches to attack person detection. In CVPR Workshops, pages 0-0, 2019.
[27] Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. arXiv preprini physics/0000077, 2000.
[28] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In IEEE Information Theory Workshop (ITW), pages 1-5, 2015.
[29] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be odds with accuracy. In ICLR, 2019.
[30] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR, 2019.
[31] Zifeng Wang, Batool Salehi, Andrey Gritsenko, Kaushik Chowdhury, Stratis loannidis, and Jennifer Dy. Open-world class discovery with kernel networks. In ICDM, pages 631-640, 2020.
[32] Chieh Wu, Zulqarnain Khan, Stratis Ioannidis, and Jennifer G Dy. Deep kernel learning for clustering. In ICDM, pages 440-648, 2020.
[33] Chieh Wu, Jared Miller, Mario Sznaier, and Jennifer Dy. Solving interpretable kernel dimen- sionality reduction. NeurlPS, 32, 2019.
[34] Ziang Yan, Yiwen Guo, and Changshui Zhang. Deep defense: Training dans with improved adversarial robustness. In NeurIPS, 2018.
[35] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference BMVC, 2016.
[36] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jor- dan. Theoretically principled trade-off between robustness and accuracy. In ICML, pages 7472-7482, 2019.
12


Reviews:
[{"review_id": "yyYtNi8RxJx", "reviewer": "Reviewer_6H63", "summary": "To enhance the adversarial robustness of deep models, this paper proposes to use HSIC bottleneck as a regularizer. Unlike previous work trained with adversarial samples, the proposed learning objective drops redundant information from the input to the latent representations. Theoretical analysis that HSIC regularizer can reduce the output variance is provided.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is valuable in the following aspects:\n1.\tThe idea that improving robustness without adversarial samples is interesting. Although HSIC bottleneck is used in previous work, it is novel to use HSIC as a regular term for adversarial robustness.\n2.\tThe theoretical analysis ensures the soundness that HSIC can reduce the influence of an adversarial attack.\n3.\tThe effectiveness is well studied with ablations, and the experiment details are clear.\n4.\tThe limitations and potential negative impacts are clearly discussed.\n\nI only find some minor issues:\n1.\tIt will be more readable if the abstract provides more information on the proposed method and theoretical conclusions.\n2.\tIt is said that SWHB fails on ResNet-18 due to the way how SWHB updates parameters. However, Ma et al. have conducted experiments on ResNet and shows comparable performance with backprop. Could you please provide more explanation?\n3.\tError bars in figure 5 are unclear. It might be better to show the standard deviations with a table.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["Privacy and Security (e.g., consent)"], "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "wadRJVDdwds", "reviewer": "Reviewer_fUk4", "summary": "This paper proposes a novel regularization for adversarial robustness based on an information bottleneck criterion. The method is well-motivated and supported theoretically and empirically.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The proposed regularization is well-motivated (Fig. 1, Lines 127-135).\n- It is theoretically shown to upper bound the variance in the network output (Theorem 1 with continuity and bounded assumptions) and the expected absolute difference of the output (Theorem 2 with Gaussian input assumption). Theorem 2 is a more direct bound for the adversarial robustness.\n- The experiments support the effectiveness of the idea in increasing the robustness of ResNet and WideResNet models on CIFAR-10 (Table 2) and CIFAR-100 (Table 1 right) when combined with other robustness methods as measured by various attacks such as AutoAttack. In particular, the improvements achieved combined with TRADES are significant (Table 1-right, CIFAR-100 ~0.5% against AA, Table 2-right, CIFAR-10 ~0.8% against AA). The improvements are shown to be statistically significant.\n\nMinor:\n- In Table 3 and Appendix E, the accuracy against PGD is low. Is this ablation study done without adversarial training? It would be better to see an ablation study on the best performing methods. Also, lambda_y is only set to 0.05 in Table 6 and a wider range needs to be studied.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact have been discussed in the appendix.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "uY_fmUcLlMW", "reviewer": "Reviewer_cTjB", "summary": "The authors propose to use an information bottleneck regularization -- with Hilbert-Schmidt information rather than mutual information -- to improve the adversarial robustness of the trained neural networks. The authors furthermore show that the minimizing HSIC between the input and the latent representation limits the variance the network output can achieve.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main idea of the paper is very nice, especially in the light of the fact that for deterministic networks the classical IB objective is infinite (cf. [2] in the paper). The HSIC functionals appear not to be suffering from this limitation. However, I have noticed several small to medium issues that need to be addressed:\n\n-) First of all, it is not exactly clear why the HSIC(Y;Z) term is required. Essentially, the classical or adversarial loss in (5) should suffice to ensure that Z contains sufficient information about Y, while HSIC(Y;Z) goes in the same direction. Moreover, in the classical IB setting, the term I(Y;Z) is usually replaced (not complemented) by the classical cross entropy loss. I have the strong feeling that it should be possible to remove this term altogether by appropriately adjusting the hyperparameters lambda_x. This is also supported by Table 3: In case HSIC(Y;Z) is missing during training, both HSIC terms turn out to be zero, indicating that the regularization on HSIC(X;Z) was too strong. This is further supported by Table 5, in which lambda_y=0 and for small lamda_x still good natural accuracy is achieved. Thus, HSIC(Y;Z) is not necessary, but a careful setting of lambda_x is required.\n\n-) Theorem 1 is extremely strong and interesting; however, I agree with previous reviewers that it is not very useful for making the intended point. In fact, for a purely Gaussian setting, I expect that similar statements can be made about I(X;Z): If Z contains only little information about X (in the sense of a conditional variance, which appears to be the case for Gaussian settings and/our HSIC measured with distance-based kernels), then also the network output can only vary little with X. Theorem 2 goes in the same direction, essentially replacing variance with the first absolute moment under a specific perturbation model. Both theorems are interesting, but they do not explicitly point at adversarial robustness, other than just pointing at general invariance to input variations.\n\n-) This brings me to a recent publication that may actually help relieving the previous issue. The conditional entropy bottleneck by Fischer replaces the I(X;Z) part in the IB functional by I(X;Z|Y). This term should indeed become zero, and setting this term to zero does not harm classification performance. If the same Theorems 1 and 2 could be formulated for HSIC(X;Z|Y) (appropriately defined), then this reduction in variation with the input would actually help make strong claims regarding generalization and adversarial robustness.\n\n-) Some of the papers in the references are still referred to via their arXiv version, despite being already published. I suggest to update the reference list.\n\nAside from that, I noticed some minor issues that can be addressed quite easily. I list them here for completeness, they do not influence the review score though:\n- [1] did not study learning dynamics, but [1] used the IB functional as a training objective. [2] did not suggest the IB functional for training, but analyzed it. [1] has a focus on supervised learning, not on auto-encoders. A reference to Fischer's conditional entropy bottleneck may be added.\n- in line 92, while y in {0,1}^k is correct, this is not standard notation; y is usually one-hot encoded.\n- after line 110, w.r.t. which distribution are the expectations taken?\n- in lines 138-140, the computational complexity also depends on d_Z; this should be acknowledged.\n- in line 171, why must the sets be closed? Is it not sufficient that they are bounded?\n- in Table 1, the caption claims that training times are shown, but I think they are missing.\n- line 244-245 contains a typo: \"with and hyperparameters\"\n- line 273 lists Shwartz et al, but I guess it should be Shwartz-Ziv and Tishby.\n- in Table 1 [ii] it is not surprising that missing the cross entropy loss yields low accuracy. Even if Z contains much information about Y, this information may not be easy to access for the remaining part of the network. This was discussed also in [2].\n\n\nedit: I acknowledge the authors' responses and change my score. Thank you!", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have partly addressed the societal impact by claiming that being too confident in adversarial robustness may be problematic. This being a method paper, I accept this statement as sufficient.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Adversarial robustness is an important problem of neural network models. This paper proposed a new regularization based on Hilbert-Schmidt Independence Criterion (HSIC) to improve adversarial robustness. The regularization consists two HSIC terms: one aims to reduce the nonlinear dependence between input and features, while the other enhances the dependence between features and outputs. Such a regularization is shown to enhance adversarial robustness in both natural training and adversarial training. The reviewers unanimously accept this paper after discussions. There are several minor points raised by reviewers that the authors promise to revise accordingly. In the preparation of final version, the authors are expected to incorporate these points suggested by the reviewers.
Decision: Accept (Poster)