Exploiting the Relationship Between Kendall's Rank Correlation and Cosine Similarity for Attribution Protection
Authors: Fan Wang
Adams Wai-Kin Kong
Venue: NeurIPS 2022 Accept (2022)

Abstract: Model attributions are important in deep neural networks as they aid practitioners in understanding the models, but recent studies reveal that attributions can be easily perturbed by adding imperceptible noise to the input. The non-differentiable Kendall's rank correlation is a key performance index for attribution protection. In this paper, we first show that the expected Kendall's rank correlation is positively correlated to cosine similarity and then indicate that the direction of attribution is the key to attribution robustness. Based on these findings, we explore the vector space of attribution to explain the shortcomings of attribution defense methods using $\ell_p$ norm and propose integrated gradient regularizer (IGR), which maximizes the cosine similarity between natural and perturbed attributions. Our analysis further exposes that IGR encourages neurons with the same activation states for natural samples and the corresponding perturbed samples. Our experiments on different models and datasets confirm our analysis on attribution protection and demonstrate a decent improvement in adversarial robustness.

Exploiting the Relationship Between Kendall's Rank Correlation and Cosine Similarity for Attribution Protection
Fan Wang Adams Wai-Kin Kong'
Schoool of Computer Science and Engineering, Nanyang Technological University Rapid-Riic Rich Object Search (ROSE) Lab, IGP, Nanyang Technological University Fan0050e.nte edu.sg sg adamskong@ntu.edd edu.
Abstract
Model attributions are important in deep neural networks as they aid practitioners in understanding the models, but recent studies reveal that attributions can be easily perturbed by adding imperceptible noise to the input. The non non-differentiable Kendall's rank correlation is key performance index for attribution protection. this paper, we first show that the expected Kendall's rank correlation is positively correlated to cosine similarity and then indicate that the direction of attribution is the key to attribution robustness. Based on these findings, we explore the vector space of attribution to explain the shorteomings of attribution defense methods using Ep norm and propose integrated gradient regularizer (IGR), which maximizes the cosine similarity between natural and perturbed attributions. Our analysis further exposes that IGR encourages neurons with the same activation states for natural samples and the corresponding perturbed samples. Our experiments on different models and datasets confirm our analysis attribution protection and demonstrate a decent improvement in adversarial robustness.
Introduction
Recently, the explainable artificial intelligence (XAI) has revived since deep neural networks (DNNs) are applied to more security-snnsitiv tasks such as medical imaging [27]. criminal justice (5) and autonomous driving 20) As one of the XAI tools, model attributions explain and measure the relative impact of each feature on the final prediction. With more non-expert practitioners being involved, it is more important for them to understand and reliably interpret the mechanism behind the outputs. Besides, EU regulators also start to enforce General Data Protection Regulation for more transparent interpretations on decision making based on Al [0] Therefore, the trustworthy attribution is becoming even more crucial.
Although numerous attribution methods have been proposed in recent studies 25
26
29
35
377, it
has been pointed out that they are vulnerable to attribution attacks. Different from standard adversarial 22 31 that focus misleading classifiers incorrect outputs, Ghorbani et al. [8] shows that it is possible to generate visually indistinguishable images which are significantly different on their attributions, but with the same predicted label. Dombrowski al. [6] emphasizes on targeted attack that manipulates the attributions to any predefined target attributions while keeping the model
attacks 3 9 18
outputs unchanged. There are also black-box attacks applied on text explanations [14] Common
ad adversarial defense mechanisms such as adversarial training [2 and distillation (23 are not able to tackle the attribution attacks; instead, researchers turn their focus on the attribution itself.
As the differences between natural and perturbed attributions are measured by Kendall's rank correlation [15), which reflects the ordinal importance among features, i.e., the proportion of order
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Original image
Perturbed image
Perturbed image
Perturbed image
cosine similarit:: 1.00 Kendall's rank correlation: 1.00
cosine similarity: 0.66
cosine similarity: 0.36
cosine similarity: 0.23
Kendall's rank correlation: 0.34
Kendall's rank correlation: 0.17
Kendall's rank correlation: D 0.09
Figure I: A visualization of integrated gradients of perturbed images by restricting & distance. The three perturbed attributions (the bottom images of the 2nd-4th columns) have the same distance (d = 0.7) to the original attribution (the bottom image of the first column). While & distance remains unchanged, Kendall's rank correlations are not guaranteed to be close. However, the cosine similarities reflect the changes of the Kendall's 's rank coefficients.
alignment of attributions between original and perturbed images, a straighttorward practice to protect the attributions against such adversaries is maximize their Kendall's rank correlation. Since Kendall's 's rank correlation is not differentiable, in previous studies, it is replaced by its differentiable alternatives, such as f dp-distance regularizers [2 4.. However, L, 4y-distance regularizers are not ideal for Kendall's 's rank correlation. As shown in Fig. we found that given fixed & --distance between original and perturbed attributions, their Kendall's rank correlations are drastically different, which indicates ε fddisstacce is unstable as measure of attribution similarity. Besides, there are also non-fp based regularizers, such as using Pearson's ''s correlation, as the surrogate measurement of Kendall's rank correlation [13, it is shown to be unstable to measure the attribution.
In this paper, we discover that cosine similarity, as a measurement emphasizing the angle between two vectors, is consistent with Kendall's rank correlation. We present a theorem stating that cosine similarity is positively correlated with the expected Kendall's 's rank correlation. Based on the discovery of angular perspective, we then explain the shortcomings of ép -nom based attribution robustness methods and propose integrated gradients regularizer (IGR), an attribution robustness training regularizer that optimizes on the cosine similarity between natural and perturbed attribution. Our further analysis shows that optimizing cosine similarity encourages neurons with the same activation states. The contributions of this work are summarized as follows:
We thcoretically show that, under certain assumptions, Kendall's rank correlation between vectors is positively correlated to their cosine similarity.
We characterize novel geometric perspective related the angles between attribution vec- tors that explains the connection between adversarial robustness and attribution robustness for attribution methods fulfilling the axiom of completeness 29
Under the angular perspective, we propose integrated gradients regularizer (IGR) to robustly train neural networks. Our method is proved to encourage neurons with the same activation states for natural and corresponding perturbed images.
The experimental results show that the proposed IGR regularizer can be embedded into adversarial training methods to improve their performance in terms of both attribution and adversarial robustness and outperform the state-of- -h-the- attribution protection methods.
The remainder of this paper is organized as follows. We first introduce the notations and previous related works in Section 2 The content starts with the theorem disclosing the relationship between Kendall's rank correlation and cosine similarity in Section 3 Based that, we discuss the vector

---

space of attribution in Section 4 and describe the proposed IGR as well as its property regarding
neuron activations in Section Section presents our experimental results and the paper concludes in Section 7
2 Preliminaries and related work
Let ((x(") y(0)) denote data points sampled from the distribution D, where x!!) I are input data and y {l,.... }} are labels. A non-bold version æ; denotes the i-th feature of vector and the capitalized version X denotes a random variable. A classifier is the mapping from input space to the logits f: _ R parameterized by 0, where fy (x) is the j-th entry of (z), and the classification result of input æ is given by the index of maximum logit ÿ = arg m Sjgn(f (z)).
2.1 Attribution methods
Model attribution, denoted by g(x), studies the importance that the input features contribute to- wards the final result. The mostly used attribution methods include perturbation-based 33 and backpropagationnbbace particular, integrated gradients (IG) [29], one of the gradient-basee methods, computes the attribution using the line integral of gradients from baseline image to the input image æ weighted by their difference, i.e.,
37
methods [T] including gradient-based attribution methods 25 26 In
g(a); (s. a.) [.²
05y(a + a(x -a)) da.
(1)
dr,
IG satisfies the axiom of completeness which guarantees g.g(z); = fy (x) fa(a). We omit the baseline image a in the later parts of this paper, and it is chosen to be a black image, i.e., O, if not specifically stated.
2.2 Attribution robustness
Recent studies reveal the vulnerability of neural networks that, similar adversarial examples, imperceptible perturbations added to natural images would have significantly different attribution while their classification results remain unchanged [.. Heo parameters instead of input images to disturb attributions and remains high accuracy on classifications. Dombrowski et ( makes targeted attack that changes original attributions any predefined attributions and gives theoretical explanation to this phenomenon.
[[2] manipulates the model
Engstrom al. 7 points that robust optimization enhances model representations and inter- pretability. Chen et al. [4] and Boopathy et al. [2] use f--orrm to constrain the distance between attributions of natural and perturbed images. Sarkar al. (24) proposes contrastive regularizer that emphasizes a skewed distribution on true class attribution while a uniform one on negative [3] directly optimizes Pearson's correlation and Singh et [2X]
class attribution. Ivankay et uses triplet loss to minimize the upper bound of the attribution distortion. Although the previous techniques present promising results, none of them exploits the angle between attributions explicitly for attribution protection. The method introduced in this work leverages the relationship between Kendall's i rank correlation and cosine similarity with theoretical support for attribution robustness.
3 Kendall's rank correlation and cosine similarity
Kendall's s rank correlation, often denoted by т, is a measurement of the ordinal relationship between two quantities, where two quantities have higher when they have more concordant pairs. Formally, Kendall's rank correlation between two vectors æ and x' can be explicitly computed by
2
Σ
sign(z; 1. signn( 2f))
T d(d -
where d is the dimension of the vectors. As Kendall's rank correlation is an important metric to quantify the differences between perturbed and natural attributions, we begin by presenting the relationship between Kendall's rank correlation and cosine similarity. It should be highlighted that all
3

---

the previous attribution robustness studies 2 13, 28 39 use Kendall's 's rank correlation as key performance index to evaluate the effectiveness of their methods.
To enhance attribution robustness, it is equivalent to force the perturbed attribution to have a higher Kendall's rank correlation with the original one. However, Kendall's rank correlation is not differentiable, it is difficult to directly optimize it. It is necessary to find an alternative that either approximates or has a consistent behavior with Kendall's rank correlation. The following theorem states that cosine similarity is an appropriate replacement as a regularization term since it is positively related to the Kendall's rank correlation (Fig. 2ap.
Theorem Given random vector Y (y), (y1,322.... 12.......................... Ma) where yE follows a positive-valued distribution, and two arbitrary vectors with the same dimension, X, X' E Rd that 24, 2 O. assume that there exists a sequence S = {X,) with X X, and X' XN, where the vectors satisfy the condition that cos(Xi,Y) 2 cos(X:+1..)) Y), and each X1+1 can be induced from its previous vector
X; through one of the following two operations,
(i) arbitrarity exchanging two entries of Xi
(ii) multiplying one entry in X; by (0,
Then Kendall's rank correlations of Y with X and X' have the property that EE [T(X, Y)] > [x(', Y)), where the expectation is taken over Y satisfying cos(X,, Y) 2 cos X; +,Y).
The full proof and discussions can be found in Appendix A In the scenario of attribution robustness, under the above assumption, we denote Y as the natural attribution g(x), and X and X' as two perturbed attributions g(z') and g(x"). If the perturbed attribution has a greater cosine similarity with natural attribution, then their expected Kendall's ' rank correlation is also greater. Explicitly speaking, if cos(g( (²), gz)) 2 (os(g(""),
then E [F(((2)) (a)a)))))))))) [(x))] 2 E [r(g(")), g()(()))))) This theorem provides a theoretical foundation that supports using cosine similarity for attribution protection because it directly links to Kendall's rank correlation.
4 Characterization of geometric perspective on attributions
In the last section, we have indicated the relationship between cosine similarity and Kendall's rank correlation. In this section, we use this relationship to explain (i) the drawbacks of attribution protections based on Ep-norm, e-g., ming (l((x) g(H)]lp in Chen et al. 4. where is natural sample and e is a perturbed sample; (ii) the inappropriateness of standard adversarial training for attribution protection and (iii) the limitation of the cosine similarity, i.e., mine (1 cos(g(#)) ), o (()))) for standard adversarial protection. In this discussion, g(x) and g()) are considered as vectors, and as stated in Theorem T- smaller angle between them implies higher attribution robustness. The attribution method g is assumed to fulfill the axiom of completeness i.e., f
(x)), V/Eg()); < E, fy (x). In other words, (19(x)) ll2 is the lower bound of (xx) and larger [[g( ((x))] implies higher classification accuracy. Thus, minimizing the angle between (xx) and ( and maximizing their magnitudes would respectively enhance their
If ((x); 20 0 for all i, (19(2))
attributional and adversarial robustness.
Fig. shows a two-dimensional projection for the ease of illustration, where each 2D point represents an attribution of an input. digensiniimli cases can be extended a similar manner. In Fig. 2b g( xz) is the original attribution of æ and the others are its perturbed counterparts. Given two
> cos(g(z). g(x")), according to Theorem [ (g(xx, g(("))())) )) is likely larger than (g(()), g(x"))
attributions, g(x") and g(x"), where [1)(x) g(x")ll = [[((z) g(a")] [I but cos(g(z),
implying that the attribution of x' is likely closer to the attribution of x than that of x". It explains the results in Fig. and point (i), i.e., drawbacks of attribution protection based on lp -noom.
The standard adversarial training maximizes f,((). where is an adversarial example. In Fig. 2b x has large [!)","")]>, implying that fy(*"") is also large. In other words, the classification label of " well protected, However, standard adversarial training does not explicitly minimize the angle between s(z) and g(x"".. It implies that (g(()), ((=")) can be small and xx can attack the attribution successfully. It should be mentioned that ad adversarial training does improve attribution
Without loss of generality, assume fa(a) 0 and use f2- the illustration.

---

0.73
Cosiue siuillrity
92
9 (
(())
0.72
g(æ)
1 g
0.71
(xx"))
0.49
Kendall's rank correlation
91
(a)
(b)
Figure 2: (a) Visualization of Kendall's rank correlation and cosine similarity using simulated data. Given fixed vector with dimension of 10,000, thousand random vectors v; sampled and their corresponding 7 and cos with y are calculated and plotted. The positive correlation can be observed and is later proved by Theorem [ (h) 21) illustration of comparison of attribution trained by no-morm and cosine similarity. The axises are two dimensions of attribution. The solid ball and dashed ball represent two networks. Solid ball represents the untrained attribution surface g and dashed ball is the trained surface g".
robustness because it smooths the decision surface explains the point (ii).
133 although it is not the most ideal one.
Point (iii) can be explained similarly. Since the cosine similarity, or ming (1 cos()( (()), does not necessarily enlarge the magnitude of (()), it cannot improve network robustness against standard adversarial attack. Fig. 2blshows two networks (the dashed circle and solid circle). cos(g(x), g cos(g(2) (0(x),g) g." (ā)). which implies that the two networks perform the same on attribution protection, but llg" (x) (l)( meaning g" is more robust against the standard adversarial attack from ø, while g is more vulnerable.
To protect against both attribution attack and adversarial attack, in the following section, the proposed IGR is optimized with adversarial loss together, where the former minimizes the angle between attribution vectors to perform attribution protection, and the latter maximizes their magnitude to offer standard adversarial protection.
Integrated gradients regularizer (IGR)
Based on the above analysis, in this section, we introduce the integrated gradients regularizer (IGR), which regularizes the cosine similarity between natural and perturbed attributions.
5.1 IGR robust training objective
Since Kendall's 's rank correlation and cosine similarity are positively related, we suggest to improve attribution robustness, especially integrated gradients, by maximizing the cosine similarity between natural and perturbed attributions, or equivalently, minimizing 1 cos(IG II()))))(((() Therefore, we propose the following training objective function incorporating the IGR
Ligr Ep(C(ă, y; 0) + A (1-
(I)) IG()))
(3)
where L is standard loss function used in robust training, and is hyper-paramette We will later show in Section that L can be chosen from existing loss function in robust training, and our IGR will further improve the robustness upon baseline methods.
In practice, the integral inside IG definition can be numerically computed by its Riemann sum, and IG is approximated by
m 0fy (#x)
iG(x); x4
m
Or;
5

---

Similar to adversarial training, optimizing the above objective function in Eq. (3) requires perturbed example ø that maximally diverts its IG from the original counterpart. Such examples can be found by maximizing the proposed regularizer within its fp-ball with radius e, L.e,
T = arg max (1 - (ZG (G()))
(5)
(x)
It is noticed that computing the adversarial loss L y; 0) itself relies on ë, which can be obtained from adversarial attacks, i.e., E = arg max; ( y: 0). Thus, here we reuse these 2 in IGR to avoid repeatedly using gradient descent methods find the optimum in Eq. (5) and speed up the training. For example, if L is the standard adversarial training loss function, we directly use the examples generated from PGD attack. The computation cost of IGR is similar to previous proposed methods.
The use of Pearson's 's correlation regularizer in Ivankay et al. [3] as the replacement of Kendall's s rank correlation is the closest method to ours. Ivankay et al. [13] suggests that Pearson's 's correlation regularizer keeps the ranking of feature constant. However, the statement is not supported by any theoretical justification while we give theorem that shows cosine similarity is positively related to Kendall's rank correlation. Besides, the Pearson's s correlation is an unstable metric for attributions with small variances. For fixed vector, two slightly different inputs 5 can have drastically different Pearson's 's correlation, which easily fluctuate from - to 1. The detailed discussion can be found in Appendix B
5.2 IGR induces more consistent neuron activation states
An interesting discovery about IGR related to neuron activations. We found that the activation functions in ReLU networks trained with IGR are more often with the same neuron activation states for natural sample and corresponding perturbed sample. For deep networks with ReLU activations, the Rre-ReLU value is positive (negative) for natural sample, the probability of Pre-ReLU value being positive (negative) for corresponding perturbed sample would increase when trained with IGR. To analyze this phenomenon, a single-layer neural network with ReLU activation is studied. The results from this single-layer neural network can be extended to deep networks by stacking multiple layers.
Recall that æ E Rd is an input image, and the network function f is parameterized by (W, c) e Rdxmm x Rm x R, where W; is the column vector of W, Waj is the ij-th entry of matrix W and u, is the i-th entry of vector u. i.e., f(x) = u ReLU(W x) + C. Then, the following proposition holds. Proposition 1. Given single-layer neural network with ReLU activation, and with the above parameterization, if. for all i, W; and u, are all independent and identically distributed random variables following Gaussian distributions, i..., W, i.i.d NO, o Ia) and un i.3dd N(0, ((0,0%) and two input images that each has small variance, I' and ă. then
IP(W T x > On Φ > 0)
IG(()) ~
(6)
P((W x O)P(W Φ > 0)
The proof can be found in Appendix A.2 The right-hand side of Eq. (6) is called the activation
consistency of natural and perturbed samples. For the sake of convenience, let the event W WT x > 0 be
A and W ã > 0 be B. The hight-hand side of Eq. (6) can be rewritten as P(A B/ P(A)P(B... Since P(An n B) has the upper bound that P(An B) min (P(4), P(B),, it is obvious that
PPAN B)
P(ANB)
P(A)P((B S (P(AB)PP(AN)
1,
and the equality holds when event A happens with the same probability event B, ie., P(A) P(B)= = (A n B); alternatively, the equality can hold when P(A9) IP(B)) P(A U B In other words, maximizing Eq. (6) encourages that the network activates the same set of neurons for x and ã, or deactivates the same set of neurons for & and R.
Experiments and results
6.1 Experimental configurations
We evaluate the performance of IGR on different datasets, including MNIST [19), Fashion-MNIST 34 and CIFAR-10 10 [17. For MNIST and MNNS,, use network consisting of four

---

Table A summary of loss functions used in AT, TRADES and MART, and added with IGR
Model
Loss function
AT (Cat)
CE((()),
TRADES (Ltrades)
CE(f( [KL((()))l(()
MART (Cmart)
BCE(f(#), ())
+IGR
+X cos
Table 2: Attribution robustness of models trained by different defense methods under IFIA (top-k).
MNIST
Fashion-MNISS
CIFAR-10
Model
top-k
Kendall
top-k
Kendall
top-k
Kendall
Standard
32.21%
0.0955
42.83%
0.1884
46.71%
0.1662
IG-NORM
36.13%
0.1562
51.84%
0.3446
74.49%
0.5811
IG-SUM-NORM 4
41.53%
0.2240
57.27%
0,4097
78.70%
0.6901
AdvAAT
13
51.74%
0.3791
73.62%
0.5810
72.11%
0.5484
ART 28
30.38%
0.1439
31.71%
0.2079
70.44%
0.6875
SSR
33
38.77%
1,1650
60.40%
0.4321
71.20%
0.5498
AT 21
34.35%
0.1846
32.00%
0.1516
72.21 %
0.5578
AT+IGR
33.40%
0.1582
53.36%
0.3750
73.37%
0.5775
TRADES 36
36.37%
0.2127
57.01
0.2582
78.28%
0.6903
TRADET+AdvAAT
52.04%
0.4315
79.15%
0.5794
71.30%
0.5239
TRADES+IGR
56.13%
0.4537
80.62
0.6565
80.26%
0.6940
MART [32
32.50%
0.1261
58.57%
0.4262
76. 16%
0.6192
MART+IGR
37.34%
0.1854
57.97%
0.4317
76.56%
0.6328
convolutional layers followed by three fully connected layers. The model is trained by Adam
Optimizer [16] with learning rate 10-4 for 90 epochs. For CiFAR-10, we train ResNet-18 [T] for 120 epochs using SGD [30] with initial learning rate 0.1, momentum 0.9 and weight decay 5 10
The learning rate decays by 0.1 at the 75th and 90th epoch. All the experiments are run on NVIDIA
GeForce RTX 3090.
As discussed in Section 5 IGR applied with state-of of the art adversarial training methods: standard adversarial training (2T) TRADES (36] and MART [32]. Lut, Ltrades and Lmart in Table [ are the objective functions of these methods, and are regarded as in Eq. (3). In Table CE denotes the cross-entropy loss and KL denotes the KL-divergence. BCE is a boosted cross-entropy (see details in Wang et al. (32)). Note that both AT and MART generate adversarial examples by maximizing CE loss, while TRADES maximizes the KI KI-divergence regularizer. Following the baseline methods, we directly leverage the perturbed examples generated by their original techniques to compute the integrated gradients, well as the IGR, instead of generating to ensure fair comparisons, we keep the apper-parameters the same for models with or without IGR.
ones using Eq. (5). Morcover,
6.2 Evaluation on attribution robustness
To evaluate our method under attribution attack, the iterative feature importance attacks (IFIA) using top-k intersection as dissimilarity function (top-k) [8] is adapted. IFIA generates perturbations by iteratively maximizing the dissimilarity function that measures the changes between attributions of images, while keeps the classification results unchanged. In this experiment, we perform 200-step IFIA as in Chen al. 4.. For MNIST and Mashion-MMISST we choose k 100 and the perturbation size 0.3. For CIFAR- k 1000 and = 8/ 8/255. Two metrics are chosen to evaluate the performance under attribution attack as in Chen et al. 4 top-k intersection and Kendall's rank correlation, where top-k intersection counts the proportion of pixels that coincide in the k most important features. Each sample attacked five times and the mean metrics are reported. For both metrics, higher number indicates that the model is more robust under the attack.
7

---

Image
Original (baseline)
Adv IG (basellne)
Original IG (+IGR)
Adv IG (+IGR)
Figure 3: IGR improves attribution robustness. The second and third images are the IG of the original and perturbed image on the baseline model. The last two images are respectively the IG of the original image and perturbed image from the model trained with IGR. Both baseline and baseline+IGR models make the correct classifications, while only baseline+IGR protects the attribution of perturbed image. More visualization results are given Appendix C
Table Adversarial accuracy (%) of CNN trained by different defense methods on MNIST, Fashion- MNIST and CIFAR-10. 10.
MNIST
Dushion-MNIST
CIFAR-10
Model
Natural
FGSM
PGD20
CW
Natural
FGSM
PGD20
CW.
Natural
FGSM
PGD20
CW
AT
99.43
99.39
99.25
99.24
89.75
78.92
74.69
74.65
73.09
69.42
37.56
45.28
+IGR
99.51
99.45
99.32
99.32
80.98
79.20
76.79
76.31
73.69
70.40
38.21
46.70
TRADES
99.40
99.36
99.21
99.19
78.82
77.66
75.94
75.58
81.33
79.15
55,02
52.40
+IGR
99.40
99.40
99.26
99.24
80.61
79.05
76,89
76.44
81.65
79.54
54.65
52.43
MART +IGR
99.39
99.29
99.09
99.08
79.43
79.36
77.91
77.49
78.97
77.19
56.05
50.99
99.51
99.39
99.28
99.24
81.51
82.13
79.93
79.01
79.27
77.35
56.47
51.11
To compare, attribution protection methods, IG-NORM and IG-SUM-NORM by Chen et al. Smooth Surface Regularization (SSR) [33), Atrributional Robustness Training (ART) (2 and Adver- sarial Attributional Training with robust training loss (AdvAAT), are implemented and evaluated on all the datasets. A cross-entropy loss trained natural model (standard) is also included a baseline. The details of these baseline methods are briefly introduced in Appendix C.2
From the results in Table 2 we observed the following phenomenons. (i) Compared with baseline methods (AT, TRADES and MART), models trained with IGR outperform their corresponding (ii) Adversarial defense
counterparts in terms of both top-k intersection and Kendall's rank correlation. methods themselves also help the attribution protection, especially improve on Fashion-MNII MNIST and CIFAR-10 comparing with the standard cross-entropy training. (ii) Compared with other attribution protection methods, standard adversarial defense methods, including AT, TRADES and MART, are weaker in attribution robustness; however, they achieve comparable protections when training with IGR. (iv) TRADES itself has the best attribution protections among models without IGR, and IGR provides the most significant boost TRADES. (v) Since AdvAAT uses Pearson's correlation as a regularizer, which is close to IGR, and TRADES+IGR outperforms the other baselines, Table for the comparison. Table Table shows clearly that TRADES+AdvAAT does not always improve over TRADES and is consistently outperformed by TRADES+IGR.
even stronger attribution
apply Pearson's correlation
TRADES, i.e., TRADES AdvAAT,, in the
visualization attribution robustness is also presented in Figure 3 It observed that the attribution of the baseline model is easily corrupted. For model trained with IGR, although the magnitudes of IG are different from IG of the original images, the directions remain nearly identical, which is also aligned with human visual perceptions.
6.3 Evaluation on white-box adversarial robustness
To evaluate the performance of IGR on adversarial robustness, the trained defense models are evaluated under white-box adversarial attacks, including FGSM (9. PGD [1] and CW> 33) where all the attacks have the information of the entire models, including architectures and parameters. The numbers are reported in both natural accuracy (Natural) and adversarial accuracies under FGSM, PGD with 20 steps (PGD20), and CW oo attacks. The maximum allowable perturbations are chosen

---

0.50
1.7
0.00
Actiration consirenccy
0.45
0.35
6.6
0.40
0.35
6.3
0.45
Baseline
0.40
TRADES
MART
TRADES
MART
TRADES
MAXI
(a) MNIST
(b) Fashion-MNIST
(c) CIFAR-10
Figure 4: Activation consistency baseline models and IGR models in MNIST, FaShion-MNIST
and CIFAR-10.
to be = 0.3 for MNIST and Fashion-MNIST, and ε 8/255 for CIFAR-I0 as previous studies. The white-box adversarial accuracy results, as well as natural accuracy are reported in Table
As shown in Table 3 defense methods trained with IGR achieve higher accuracies under all three types of attacks upon their corresponding baseline methods, except TRADES+IGR under PGD attack in CIFAR-10. 10. In the meantime, classification accuracies of natural images are also improved in seven of nine evaluations. This suggests that training with IGR improves adversarial accuracies without losing the generalization of natural accuracies. Although IGR is designed for attribution protection, these improvements considered as a side-effect and rigorous study of the phenomenon can be future work.
6.4 Evaluation on activation consistency
This section reports the experimental results that verify the claim in Section 5.2 IGR encourages that the network activates the same set of neurons for natural and perturbed samples æ and ã. During the experiments, all the pre-activation values recorded and used compute the proportion of nonnegative values. Thus, the activation consistency defined on the right-hand side of Eq. (6) can be numerically computed.
Fig. f compares the activation consistency on the baselines and the corresponding models trained with IGR. It is noticed that for all the datasets, the activation consistency of the models trained with IGR are consistently greater than the corresponding baseline models, which verifies our theory in Section [.2] Moreover, reported in Table 2 the improvements of AT+IGR in MNIST and MART+IGR in Fashion-MNIST are not as significant as others. The results are also reflected on activation consistency, the value of activation consistency slightly improves from 0.40 to 0.43 on AT+IGR MNIST and from 0.67 to 0.69 on MART+IGR Fashion-MNIST, while TRADES+IGR that boosts the most in attribution robustness also increases the most in activation consistency.
Conclusions
In order to leverage the non-differentiable Kendall's rank correlation for attribution protection, this paper starts with a theorem indicating the positive correlation between cosine similarity and Kendall's s rank correlation. We then introduce geometric perspective to explain the shortcomings of lp based attribution defense methods and propose the integrated gradients regularizer to improve attribution robustness. It is discovered that IGR encourages networks activating the same set of neurons for natural and perturbed samples. Finally, experiments show that IGR can be combined with adversarial objective functions, which simultaneously minimizes the angle between attribution vectors for attribution robustness and maximizes their magnitude to offer standard adversarial protection.
Acknowledgments and Disclosure of Funding
This work is partially supported by the Ministry of Education, Singapore through Academic Research Fund Tier 1, RG73/21

---

References
[1] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Rober Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7))e0130144 2015.
[2] Akhilan Boopathy, Sijia Liu, Gaoyuan Zhang. Cynthia Liu, Pin- Yu Chen, Shiyu Chang, and Luca Daniel. Proper network interpretability helps adversarial robustness in classification, In International Conference on Machine Learning, pp. 1014-1023. PMLR, 2020.
[3] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEEE Computer Society, 2017.
[4] Jiefeng Chen, Xi Wu, Vaibhav Rastogi, Yingyu Liang, and Somesh Jha. Robust attribution regularization. In Advances Neural Information Processing Systems, pp. 14300-14310, 2019.
[5] Ashley Deeks. The judicial demand for explainable artificial intelligence. Columbia Law Review, 1197):1829-1850, 2019.
[6] Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus- Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. In Advances Neurul Information Processing Systems, pp. 13589-13600, 2019.
[7] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness prior for learned representations. arXiv preprint arXiv:1906.00945, 2019.
[8] Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 368 3681-3688, 2019.
[9] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. In International Conference on Learning Representationss 2015.
[10] Bryce Goodman and Seth Flaxman. European union regulations on algorithmic decision-making and a "right to explanation". Al magazine, 38(3):50-577 2017.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
[12] Juyeon Heo, Sunghwan Joo, and Taesup Moon. Fooling neural network interpretations via adversarial model manipulation. In Advances in Neural Information Processing Systems, pp. 2925-2936, 2019.
[13] Adam Ivankay, Ivan Girardi, Chiara Marchiori, and Pascal Frossard. Far: general framework for attributional robustness. arXiv preprint arXiv:2010.07339, 2020.
[14] Adam Ivankay, Ivan Girardi, Chiara Marchiori, and Paseal Frossard. Fooling explanations in text classifiers. arXiv preprint arXiv:2206.03 2022.
[15] Maurice George Kendall. Rank correlation methods. 1948.
[16] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 11Xiv:1412.66980. 2014.
[17] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[18] Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial machine learning scale. In International Conference on Learning Representations, 2017. URL htt./https:///pereeeei
nett/forumiiddBBJAAT4
[19] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://ann.lcuu..co//etttt 2, 2010.
10

---

[20] Maria Paz Sesmero Lorente, Elena Magán Lopez, Laura Alvarez Florez, Agapito Ledezma Espino, José Antonio Iglesias Martinez, and Araceli Sanchis de Miguel. Explaining deep learning-based driver models. Applied Sciences, 118)3332,, 2021.
[21] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. In International Conference on Learning Representations, 2018. URL httpt:t:/ppperrviiieeeee/// http:/eeefforrue pttp:/forummiddxxIIBBB..
[22] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 20/6 IEEE European symposium on security and privacy (EuroS&P), pp. 372-387. IEEE, 2016.
[23] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation defense to adversarial perturbations against deep neural networks. In 2016 IEEE symposium on security and privacy (SP), pp- 582-597. IEEE, 2016.
[24] Anindya Sarkar, Anirban Sarkar, and Vineeth N Balasubramanian. Enhanced regularizers for attributional robustness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 2532-2540, 2021.
[25] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, pp. 3155 -3533 PMLR, 2017.
[26] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16 16, 2014, Workshop Track Proceedings, 2014.
[27] Amitojdeep Singh, Sourya Sengupta, and Vasudevan Lakshminarayanan. Explainable deep learning models in medical image analysis. Journal of Imaging, 6(6):52, 2020.
[28] Mayank Singh, Nupur Kumari, Puneet Mangla, Abhishek Sinha, Vineeth N Balasubramanian, and Balaji Krishnamurthy. Attributional robustness training using input-gradient spatial align- ment. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt XXVII 16, pp. 515-533, Springer, 2020.
[29] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pp. 3319-3328. PMLR, 2017.
[30] Iya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp. 1 1139-1147 PMLR, 2013.
[31] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, ICLR 2014, 2014.
[32] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International Conference on Learning Representations, 2019.
[33] Zifan Wang, Haofan Wang, Shakul Ramkumar, Piotr Mardziel, Matt Fredrikson, and Anupam Datta. Smoothed geometry for robust attribution. In Advances in Neural Information Processing Systems, volume 33, pp. 13623-13634 Curran Associates, Inc., 2020.
[34] Han Xiao, Kashif Rasul, and Roland Vollgraf. benchmarking machine learning algorithms. arXiv preprini arXiv: I 1708.077777 2017.
Fashion-mnist: novel image dataset for
[35] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818- 833. Springer, 2014.
[36] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. International Conference on Machine Learning, pp. 7472-7482, PMLR, 2019.
11

---

[37] Luisa M. Zintgraf, Taco S. Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. In International Conference on Learning Rep- resentations, ICLR 20/7. OpenReview.net, 2017. URL. bttp:ttts:///oeeeeee http///ppereiie...eoe/// wtwpa/rriiieeenteoorrn.e. id=BJ5UeU9xx
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [N/A]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "_8c-fi9YDPW", "reviewer": "Reviewer_zTbg", "summary": "This paper theoretically demonstrates that Kendall’s rank correlation of two vectors is correlated to their cosine similarity. Then the authors propose an integrated gradients regularizer (IGR) to regularize the adversarial training methods and make them more robust.", "questions": "Could the cosine similarity between integrated gradients be used to select harder adversarial examples during training?", "limitations": "No limitation is submitted.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors demonstrate the relationship of two correlation metrics Kendall’s rank correlation and cosine similarity theoretically and intuitively. \n2. This paper is well-written and well-organized.\n3. The integrated gradient regularizer helps adversarial training achieve more improvements.\n4. The integrated gradients regularizer can make the model more robust.\n\nWeaknesses:\n1. Some experimental settings should be described more clearly, such as the top-k metric.\n2. No limitations are claimed in the paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9AGJe8oaqJX", "reviewer": "Reviewer_H33g", "summary": "This paper proposes integrated gradient regularizer (IGR) for attribution protection. Incorporating IGR is to combine a standard (adversarial) loss function with a similarity function measuring the difference between the natural and perturbed attributions. The authors propose to use the cosine similarity of the integrated gradients (IG) as the similarity function, and they theoretically show that it is an appropriate differentiable replacement for Kendall’s rank correlation, which is a metric quantifying the differences between attributions. The authors also show the downsides of previous attribution protection methods based on $l_p$-norm, along with the shortcomings of standard adversarial training for attribution protection and cosine similarity for adversarial protection. Experiments show that IGR can both contribute to attribution protection and adversarial protection, surpassing previous baselines.", "questions": "In the caption of Figure 2, the authors say that “Solid ball represents the untrained attribution surface $g$ and the dashed ball is the trained surface $g^*$.” What do “trained” and “untrained” mean?", "limitations": "I suggest the authors describe the concept of attribution protection in more detail in the preliminaries, and emphasize its significance, for the concept seems to be not very popular in the ML community.\nI suggest the authors make a box plot of +IGR and baseline models to show the variance on multiple samples.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe proposed IGR is simple and effective, strengthening both adversarial and attribution robustness. The advantages are shown both theoretically and empirically. The presentation is generally clear.\n\nWeaknesses (and suggestions):\n\n- The title is too narrow. I suggest not to mention the specific metric “Kendall’s Rank Correlation”.\n- The detail of choosing $\\lambda$ is missing. What value is set for it and how does it affect the results?\n- In section 4, I suggest the authors first point out the implications of (a) the angle between the attribution vectors and (b) their magnitudes, and state the importance of minimizing (a) as well as maximizing (b). When explaining the drawbacks, I suggest the authors focus on the characteristics of the methods for minimizing (a) and maximizing (b).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "08VOZYqHtK7", "reviewer": "Reviewer_3p2D", "summary": "### Summary\n\nThe authors study an important problem in the space of feature attribution robustness. Based on the cited works in the paper, feature attribution methods are vulnerable to adversarial attacks, such that is possible to obtain remarkably  different attribution by adding an adversarial noise to the input without changing the output label of the model (in the case of classification). This begs the question if it is possible to protect the model against such attribution attacks.  The paper highlights some of the weaknesses in existing attribution robustness (attribution defense) methods such as l_p distance regularizers and Pearson correlation. The authors make a key observation about the Kendall's rank correlation metric which is a commonly used metric for evaluation feature attribution robustness and propose a differentiable alternative based on cosine similarities which can be embedded during training.\n\nThe main contributions of the are as follows.\n- Theoretically showing the relationship between Kendall's rank correlation between natural and perturbed attribution and their cosine similarity (under certain assumptions).\n- Providing a geometric perspective to explain connection between adversarial robustness and attribution robustness\n- Proposing a novel Integrated Gradients Regularizer method which (i) Improves attribution robustness and (ii) increases activation consistency for natural and perturbed examples.\n\nOverall it is a good paper and I recommend the paper for an accept. \n\n\n", "questions": "### Questions\n\n- [ ] I have not been able to read the prior work in full detail. Can you please clarify if both the prior work https://arxiv.org/pdf/2010.07393.pdf  and the current work seem to involve an outer minimization and an inner maximization to obtain the adversarial perturbed example ? Is this prior work differentiable ?\n- [ ] Can you include AdvAAT loss function in Table 1 and highlight the difference between AT and AdvAAT. I think I understand the difference based on going through prior work but would be good to confirm.\n", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths and Weaknesses\n\n\nStrengths:\n- [ ] Providing a theoretical justification for using cosine similarity and exploiting its relationship with kendall rank correlation metric which seems to not have been done in the prior work in https://arxiv.org/pdf/2010.07393.pdf .\n- [ ] Theoretical proof in the Appendix around instability of pearson correlation for small variance. \n- [ ] The geometrical perspective with associated figure and explanation is convincing and intuitively explains the claims.\n\nWeakness:\n- [ ] One major concern I have with the paper is that while theoretically it is shown the Pearson correlation is unstable and the IGR (Integrated Gradient Regularization) method seems to give best performance, it would also be helpful to compare the it's correlation with Kendall rank correlation. i.e. a figure similar to Figure 2(a) but for the pearson correlation term used in prior work.\n- [ ] Suggested though not important for review decision: Some discussion around how the KL divergence term in TRADES might be interacting with IGR to give large gains.\n\nnit:\n- [ ] Line 103: Typo paris -> pairs\n- [ ] Suggest to cite the following in Related Works\n\t- [ ] https://arxiv.org/pdf/2206.03178.pdf\n\t- [ ] https://www.aaai.org/AAAI22Papers/AAAI-3836.HuaiM.pdf\n\t- [ ]\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept