Robust Explanation Constraints for Neural Networks
Authors: Matthew Robert Wicker
Juyeon Heo
Luca Costabello
Adrian Weller
Venue: ICLR 2023 poster (2023)
Keywords: Explainability
Adversarial Robustness
Neural Networks
Robustness Certification

Abstract: Post-hoc explanation methods are used with the intent of providing insights about neural networks and are sometimes said to help engender trust in their outputs. However, popular explanations methods have been found to be fragile to minor perturbations of input features or model parameters. Relying on constraint relaxation techniques from non-convex optimization, we develop a method that upper-bounds the largest change an adversary can make to a gradient-based explanation via bounded manipulation of either the input features or model parameters. By propagating a compact input or parameter set as symbolic intervals through the forwards and backwards computations of the neural network we can formally certify the robustness of gradient-based explanations. Our bounds are differentiable, hence we can incorporate provable explanation robustness into neural network training. Empirically, our method surpasses the robustness provided by previous heuristic approaches. We find that our training method is the only method able to learn neural networks with certificates of explanation robustness across all six datasets tested.  

Published as a conference paper at ICLR 2023
ROBUST EXPLANATION CONSTRAINTS FOR NEURAL NETWORKS
Matthew Wicker*!l, Juyeon Heo*. Luca Costabello Adrian Weller!.2
The Alan Turing Institute, 2 University of Cambridge, AAcenttree Labs
ABSTRACT
Post-hoc explanation methods are used with the intent of providing insights about neural networks and are sometimes said to help engender trust in their outputs. However, popular explanations methods have been found to be fragile to minor perturbations of input features or model parameters. Relying on constraint relax- ation techniques from non-convex optimization, bounds the largest change an adversary can make to gradient- based explanation via bounded manipulation of either the input features model parameters. By propagating a compact input or parameter set as symbolic intervals through the forwards and backwards computations of the neural network we can formally cer- tify the robustness of gradient- maddent-bbsed explanations. Our bounds are differentiable, hence we can incorporate provable explanation robustness into neural network training. Empirically, our method surpasses the robustness provided by previous heuristic approaches. We find that our training method is the only method able to learn neural networks with certificates of explanation robustness across all six datasets tested]
develop method that upper-
INTRODUCTION
Providing explanations for automated decisions is principal way to establish trust in machine learn- ing models. In addition to engendering trust with end-users, explanations that reliably highlight key input features provide important information machine learning engineers who may use them aid in model debugging and monitoring (Pinto et al. 2019 Adebayo et al. 2020 Bhatt et al. 2020a b). The importance explanations has led regulators considering them as a potential requirement for deployed decision-making algorithms (Gunning 2016, Goodman & Flaxman 2017p. Unfor- tunately, deep learning models can return significantly different explanations for nearly identical inputs (Dombrowski et al. 2019) which erodes user trust in the underlying model and leads to pes- simistic outlook on the potential of explainability for neural networks (Rudin) 2019). Developing models that provide robust explanations, i.e., that provide similar explanations for similar inputs, is vital to ensuring that explainability methods have beneficial insights (Lipton) 2018).
Current works evaluate the robustness of explanations in adversarial setting by finding minor manipulations to the deep learning pipeline which cause the worst-case (e.g., largest) changes to the explanation (Dombrowski et al. 2019, Heo et al., 2019). It has been shown that imperceptible changes to an input can fool explanation methods into placing importance on arbitrary features (Dombrowski al.) 2019: Ghorbani et al. 2019) and that model parameter can be manipulated to globally corrode which features are highlighted by explanation methods (Heo et al. 2019). While practices for improving explanation robustness focus on heuristic methods avoid current attacks Dombrowski et al. 2022 Wang et al., 2020 Chen et al. 2019). it is well-known that adversaries can develop more sophisticated attacks to evade these robustness measures (Athalye et al.) 2018). To counter this, our work establishes the first framework for general neural networks which provides a guarantee that for any minor perturbation to a given input's features or to the model's s parameters, the change in the explanation is bounded. Our bounds are formulated over the saput-gradient of the model which is a common source of information for explanations (Sundararajan et al.] 2017] Wang Our guarantees constitute a formal certificate of robustness for neural network's
al. 2020
Author
email addresses in listed order: mwicker@turing.ae jh2324@ cam.ac.uk,
hica.costabello@ aww.eeelll@@cccettuee..o.. aweller@uuring.o..u

---

Published as a conference paper at ICLR 2023
explanations at given input which can provide users, developers, and regulators with heightened sense of trust. Further, while it is known that explanations of current neural networks are not robust, the differentiable nature of our method allows to incorporate provable explanation robustness as a constraint at training time, yielding models with significantly heightened explanation robustness.
Formally, our framework abstracts all possible manipulations an input's features model's pa- rameters into a hyper-rectangle, a common abstraction in the robustness literature (Mirman et al.
2018 Gowal et al. 2018). We extend known symbolic interval analysis techniques in order to propa- gate hyper-rectangles through both the forwards and backwards pass operations of a neural network. The result of our method is a hyper-rectangle over the space of explanations that is guaranteed to contain every explanation reachable by an adversary who perturbs features or parameters within the specified bounds. We then provide techniques that prove that all explanations in the reachable expla- nation set are sufficiently similar and thus that successful adversarial attack exists. Noticing that smaller reachable explanation sets imply more robust predictions, we introduce a novel regulariza- tion scheme which allows us minimize the size of of the explanation set during parameter inference. Analogous to state-of-the-art robustness training (Gowal et al. 2018). this allows users to specify input sets and parameter sets explainability constraints at train time. Empirically, framework on six datasets of varying complexity from tabular datasets in financial applications to medical imaging datasets. We find that our method outperforms state-of-the-an art methods for improv- ing explanation robustness and is the only method that allows for certified explanation robustness even on full-color medical image classification. We highlight the following contributions:
test our
We instantiate framework for bounding the largest change to an explanation that based on the input gradient, therefore certifying that no adversarial explanation exists for a given set of inputs and/or model parameters.
We compute explicit bounds relying on interval bound propagation, and show that these bounds can be used to regularize neural networks during learning with robust explanations constraints.
Empirically, our framework allows us to certify robustness of explanations and train networks with robust explanations across six different datasets ranging from financial applications to medical image classification.
2 RELATED WORK
Adversarial examples, inputs that have been imperceptibly modified to induce misclassification are well-known vulnerability of neural networks (Szegedy al. 2013 Goodfellow et al., 2015 Madry
et al. 2018). A significant amount of research has studied methods for proving that no adversary
can change the prediction for given input and perturbation budget Tieng et al. 2017 Weng et all Fazlyab et al., 2019). More recently, analogous attacks on gradient-based explanations have been explored (Ghorbani et al. 2019 Dombrowski et al. 2019 Heo et al. 2019). In (Ghorbani et al., Dombrowski et al. (019)) the authors investigate how to maximally perturb the explanation
2018
2019
using first-order methods similar to the projected gradient descent attack on predictions (Madry
al. 2018). In (Heo et al. 2019: Slack al., 2020, Lakkaraju & Bastani] 2020 Dimanov all the authors investigate perturbing model parameters rather than inputs with the goal of finding model that globally produces corrupted explanations while maintaining model performance. This
2020
attack on explanations has been extended to a worrying use-case of disguising model bias @Dimnnvv
et al. 2020
Methods that seek to remedy the lack of robustness work by either modifying the training procedure (Dombrowski et al. 2022 Wang et al. (020) or by modifying the explanation method (Anders al.)
2020 Wang et al.) 2020 Si et al. (021). Methods that modify the model include normalization of
the Hessian norm during training in order to give a penalty on principle curvatures (Dombrosski
al. 2022 Wang al. 2020). Further work suggests attributional adversarial training, searching
for the points that maximize the distance between local gradients (Chen et al., 2019: Ivankay et al.
2020 [Sinhh et al.] 2020 Lakkaraju al.) 2020). Works that seek to improve explanation in
model agnostic way include smoothing the gradient by adding random noise (Smilkov et al. 2017
Sundararajan al.
2017) by using ensemble of explanation methods (Rieger & Hansen 2020
The above defenses cannot rule out potential success of more sophisticated adversaries and it is well-known that the approaches proposed for improving the robustness of explanations do not work
2

---

Published as a conference paper at ICLR 2023
against adaptive adversaries in the case of prediction robustness (Athalye et al. 2018 He et al. 2017). However, certification methods have emerged which provide sound guarantees that even the most powerful adversary cannot fool models prediction (Wicker et al. 2018 Gehr et al. 2018 Wu et al. 2020). Moreover, these methods have been made differentiable and used for training mod- els with provable prediction robustness and for small networks achieve state-of-the-ar robustness results (Gowal et al. 2018 Mirman et al., 2018 Wiikkr et al. [022I. This work extends these certi- fication methods to the problem of robust explanations. By adapting results from symbolic interval analysis of Bayesian neural networks (Wicker et al. 2020; Berrada et al. 2021p and applying them to interval analysis of both the forward pass and backwards pass we are able to get guarantees on the robustness of gradient-based explanations. Appendix summarizes literature on non-gradient-basee explanations.
BACKGROUND
We consider neural networks tasked with solving a supervised learnin problem where we are given a dataset of n--maayy inputs and labels, D (x(1), ( } with inputs 2x(1) E R", and corre- sponding target outputs yft) E I!' either one-hot class vector for classification or real-valued vector for regression. We consider a feed forward neural network (NN) as a function fr R." - R" parametrised by a vector 0 E R2 containing all the weights and biases the network. Given a NN S composed of K layers, we denote by f8.1 J pe.k the layers of l and we have that the param- eters are given by the set, 0 = ([W((0) JKm)) U ((6()) where Wi( and ((0) represent weights and biases of the i-th layer of f Moreover, we take a to be the activation function of the network and C to be the loss function. Given this, we can write down the forward and backwards pass w.r.t
an input through a network network as follows:
Backward Pass:
Forward Pass:
kkk) _./oo_())
(0)
8(h-1) 022(h) aQ(0)
(()) ((())(t-11) g00
acwi) Oz(h-1)
(()) (H))
d = (t)) o ((k-1)
where @ represents the Hadamard or element-wise product. We highlight that d = 0L/0r, the gradient of the loss with respect to the input, and will refer to this with the vector v, i..., v = Vf (z)). The input gradient tells us how the NN prediction changes locally around the point and is therefore a primary source of information for many explanation methods (Sundararajan et al,
2017
Zintgraf et al. 2017. Smmlkov et al. 2017).
3.1
ATTACKS ON EXPLANATIONS
Input Perturbations Analogous to the standard adversarial attack threat model, targeted input perturbations that attack explanations seek to modify inputs achieve a particular explanation. We denote an adversary's 's desired explanation vector as targ
E IR... The goal of the adversary is to find an input adv such that h(2 mdv) S T where yadv = V معب (a-tta")))) and is a similarity threshold that determines when the attack is sufficiently close to the target. The structural similarity metric, Pearson correlation coefficient, and mean squared error have all been investigated as forms of h (Adebayo et al. 2018). We use the mean squarred error for h throughout this paper and discuss cosine similarity in Appendix H Moreover, for the untargeted attack scenario, an attacker simply wants to corrupt the prediction as much possible. Thus no target vector is set and an attack is successful if h(, rddr) 2 T where 7 is a threshold of dissimilarity between the original explanation and the adversarially perturbed explanation.
is similarity metric,
Model Perturbations Not typically relevant when attacking predictions is the model parameter perturbation setting. In this setting, an attacker wants to perturb the model in way that maintains predictive performance while obfuscating any understanding that might be gained through explana- tions. As before, we assume the adversary has a target explanation targ and that the adversary is successful if they find a parameter gulv such that h(r" galt) < T where adv = V (z)). The untargeted model attack setting follows the same formulation as in the input attack setting.
3

---

Published as a conference paper at ICLR 2023
Model parameter attacks explanations usually global and thus one tries to find single value of gudv that corrupts the model explanation for many imputs at once. This has been investigated in (Heo et al. 2019; Slack et al. 2020 Lakkaraju & Bastani 2020 and in (Anders et al.] 2020)]
Dimanov et al. 2020 where the authors explore modifying models to conceal discriminatory bias.
4 PROBLEM STATEMENTS
In this section, we formulate the fragility (the opposite of robustness) of an explanation as the largest difference between the maximum and minimum values of the input gradient for any input inside of the set 7'' and/or any model inside the set M. We then show how one can use the maximum and minimum input gradients to rule out the existence of successful targeted and untargeted attacks. We start by formally stating the key assumptions of our method and then proceed to define definitions for robustness. Assumptions: the methods stated the text assume that input sets and weight sets are represented as intervals, we also assume that all activation functions are monotonic, and that all layers are affine transformations e.g., fully connected layers or convolutional layers. We discuss when and how one can relax these assumptions in Appendix A Now, we proceed define what it means for a neural network's ss explanation to be robust around an input x E R".
Definition Input-Robust Explanation Given a neural network fo, test input x, a loss function C. compact input set T, and vector â E R", we say that the explanation of the network f is 5- 5-input-robinn robust iff e [n]:
min OL TET dx,
OL TET dr;
max
S 6;
Moreover we define the set Frr (v' Vi E [n], min 0c/0r; S 1/ S max,er ac/Or,) to be
the reachable set of explanations.
An intuitive interpretation of this definition is as a guarantee that for the given neural network, input, and loss, there does not exist an adversary that can change the input gradient be outside of the set Er, given that the adversary only perturbs the input by amount specified by T. Further, the vector d defines the per-feature feature-dimension fragility (lack of robustness). Under the interval assumption, we
take T to be [a"",""] such that Vi E [n]. x < I; I U7 For simplicity, we describe this interval with a width vector e E IR" such that [x x²' = [x e, + e]. We now present the analogous definition for the model perturbation case.
Definition 2 Model-Robut Explanation Given a neural network fo, test input loss function L. a compact parameter set M, and a vector E RR" we say that the explanation of the network f is 8 5-model-ronee roouutt iff [n]]: |[minges OL/Oz; mmxxeM 0C/Oz;1l 5,. Moreover, we define EM = {v' Iii E [m], mingeM OC/Or; < of < maxseM 0L/Oz.).
This definition is identical to Definition | save for the the fact that here the guarantee over the model parameters. As such, model that satisfies this definition guarantees that for the given neural network and input, there does not exist an adversary that can change the explanation outside of E& given that it only perturbs the parameters by amount specified by M. Similar to what is expressed for inputs, we express M as an interval over parameter space parameterized by a width vector E E R.a To avoid issues of scale, we express interval over weight vector W w.r.t. y as [W- -y W| 1, W + >/]] Before proceeding we remark on the choice of L. Typically, one takes the input-gradient of the predicted class be the explanation; however, one may want generate the class-wise explanation for not just the predicted class, but other classes as well (Zhou et al.) 2016 Selvaraju et al. 2017). In addition, adversarial robustness literature often considers attacks with respect to surrogate loss functions as valid attack scenarios (Carlini & Wagner 2017). By keeping L. flexible in our formulation, we model each of these possibilities.
4.1 FROM ROBUST EXPLANATIONS TO CERTIFIED EXPLANATIONS
In this section, describe the procedure for going from 5-robustness (input model robustness) to certification against targeted or untargeted attacks. For ease of notation we introduce the nota- tion n for the maximum gradient (max V, (.L(((x)) or maxsem Vf (x))) and of for
4

---

Published as a conference paper at ICLR 2023
V 0.20 6.l1
Gi 9.09
0.00
In 6.075
U.S. 0.20
b01 to 00.
0.051
0.0 0.0 0.13
Complex
Cortifiblly
Unressing
By increasing the parameters of our method, we are able to enforce certifiable locally- linear behavior of an NN model on the half-moons dataset, training data visualized in the top left corner. In the extreme case, illustrated on the far right, we recover globally linear classifier. The top row varies é the size of the input region while the bottom row varies Y which enforces locally linear behavior w.r.t. the model parameters.
Figure I:
the minimum gradient (minger (z) or mingeM V C(f(((z)) (z))) This notation leads to 5 |0 1 and E = [o o according Definitions 1 and 2 Below discuss how to use these values to certify that no successful targeted or untargeted attack exists.
Certification of Robustness is sufficient to check that Ve' E E, (((v, ( > т. In order to do so, we can define y == arg min,e h(u, ). If the inequality h(yoan ylar) successful adversary exists. Where / is the mean squared error similarity, we can construct e from the E' that maximizes the similarity by taking each component if be exactly equal to 2i when ii E [u.", "."), we take us = of when v pang > of and we take ufen v." otherwise. This construction follows from the fact that minimizing the mean squared error between the two vectors accomplished by minimizing the mean squared error for each dimension.
Targeted Attacks: To certify that successful targeted attack exists
holds, then have proven that no
Certification of Robustness to Untargeted Attacks: Similar to the argument above, we can certify the robustness to untargeted attacks by showing that Ve' E, h(v, v') S where the original input gradient. As before, if the ب ;= arg m h(v, v²) has mean squared error less than then have certified that no successful untargeted attack exists. We construct yeen this case by taking each component ufen uf when |u!" Ual |0/ o/'1 and tf = vf otherwise.
COMPUTATIONAL FRAMEWORK
In this section will provide computationally efficient framework using interval bound propaga- tion (IBP) to compute values for uL and ي We highlight that the values we compute are sound but not complete. In short, this means the values for v computed here not the largest values for the minimum that can be obtained and the values for v6 are not the smallest values for the maximum that can be obtained. Tighter bounds may be achieved the cost of greater computational complexity as discussed in Appendix A however the method we have chosen has computational cost equivalent to four forward passes through the network and is therefore very fast. order to compute sound v and D we propagate input and/or model intervals through the forward and backwards pass of the neural network; however, standard propagation techniques not accommodate joint propagation of input and weight intervals which is necessary for the backwards pass. To solve this problem, we present Lemma
Given an interval over matrices A", A" E Raxb s.t. Vi.j it holds that A S. AV and an interval over matrices B². B Bbxe s.t. Vi,j B¹ < BU We denote the center and width of a matrix interval as B = (BL + B/2/2 and B" = (BU B²)/2. We also denote M B B" and correspondingly M B)) We then have the following element-wise inequalities AAA r E [A+, [A] and vB= E BBL B²):
Lemma 1 Wicker 2021
Vi,j A MB M Qis < A B S A M M + Qis
5

---

Published as a conference paper at ICLR 2023
where 1.1 indicates element-wise absolute value and Q |A> We denote function that returns these matrix nultiplication upper and lower bounds given the four parameters above as: C(A", B") the lower bound and (A', A BB) for the upper bound.
As desired, Lemma | allows us to jointly propagate intervals over inputs and parameters through affine transformations. We can now restate the forwards and backwards pass in terms of lower and upper bounds on input features and model parameters:
Backward Pass w.r.t. Bounds:
Forward Pass Bounds: 2L.00) x² 2(0.0) x.
(L.,)) -01 /12/11.00) d(u -0
gea (z(1.k) 0z(1.)) (c(1,.) (C(uA)
(((nkt)) c(W(t.)) w((l))) g(tn) , (00))
ae(a) (ç(0) Oz(i-1)) 02(k-1)
((Uik+1) u(WiLL,)) willd), 2(2, (0))))
=1 02(L,k) )z((.h) oc(L.k) (g(UAK)
2(2,8) = a(C(enA) (1 tbll.)))
Og(R) (a(h) 0z(k-1)' 02(1-1)
d 1) -MaxMul(a dlaik) g(W,k-1), 8(118-11)
( (( (0 tolla
d -MinMul(d allaa) glesa ((l.
where B. {AB, AD, BC, BD} and the MinMul is defined analgously for the elementwise minimum. Fol- lowing the above iterative equations we arrive at d4.0 (lower bound) and du,00 (upper bound) which and for all parameters are upper and lower bounds on the input gradient for all inputs in [xc |c=,="] and
is short-hand for the elementwise maximum of the products
in [0L, 88") Formally, these bounds satisfy that Vi E [n], d/-0 Su and d. 2 of We provide more detailed exposition and proof in Appendix Finally, Exent = Id du.0)) which satisfy the conditions of Definition and
can write goert du.0
d and
5.1 GRADIENT CERTIFICATION TRAINING Loss
It follows from Definitions and that models with smaller values are more robust. To minimize 5 during training, it is natural to add Σx= 6, as a regularization term to the network loss. At training time. we consider the input perturbation budget 6 as well as the model perturbation budget n. The input interval is simply taken to be [z E, + Ed] and the interval over weight parameter W is [W- |W m. W + |Whrs) Given such intervals, we can then state a function [ar, 0, <t; 7t)) which returns the vector Σe= 8şem.. Thus, minimizing ((z, 8, ct. (47%) corresponds to minimizing an upper bound on the maximum difference in the first derivative leading to improved certification. The complete, regularized loss which we term the gradient certification (Grad. Cert.) loss is:
Lirad Cent. (x) + a D 6t, 71))
where is a parameter that balances the standard loss and the adversarial loss. Throughout the paper, we will continue to refer to the values of and that are used at training time as and The optimal loss for our regularization term is achieved for an input point æ when D(z, 6, <t, 7c) 0. For input robustness, implies that for all points in set T the gradient with respect to the input is constant. Thus, one perspective on our regularization is that it enforces certifiably locally linear behavior of the neural network for all points in T. For " it is similar guarantee save that all of the models in M must have point-wiss wise linear behavior at x. In Figure | we visualize the effect of taking regularization term to the extreme in planar classification problem. We see that indeed this perspective holds and that increasing the parameters 6 or "/t leads to the classifier having more linear behavior.
EXPERIMENTS
In this section, we empirically validate our approach to certification of explanation robustness as well as our regularization approach to training NNs with certifiably robust explanations. We start by covering the evaluation metrics we will use throughout this section to benchmark robustness of explanations. We analyze robustness of explanations on two financial benchmarks, the MNIST hand- written digit dataset, and two medical image datasets. We report exact training hyper-pate sneer-prramterrs and and discussion
details in the Appendix along with further ablation studies, Appendix of assumptions and limitations in Appendix A

---

Published as a conference paper at ICLR 2023
ADULT
Fardded
Se scard
023
0.7
Figure 2: Top Row: Only training with gradient certified loss gives non-trivial certified robust- ness performance (bars labelled Certified Robustness). We find no drop in accuracy when using our method (bars labelled Accuracy), and in addition have attack robustmess that surpasses bascline models (bars labelled Attack Robustness). Bottom Row: We introduce a variable amount of label poisoning (x-axis) to introduce bias into the datasets, and then test to see if the explanation indicates that it is using sensitive features. Looking at the magnitude of the gradients, we observe that gradient certified training forces the model to reveal its sensitivity much more readily.
Evaluation Metries In this section we state the metrics that we will use throughout our analysis. Each of the metrics below are stated for set of test inputs (-00)) Pi=0 and for each application we
will explicitly state the used.
Attack robustness: Given a neural network f we denote adv as the input gradient resulting from an attack method on the test input ((t) We then define the metric as: 1/j EVD [ (()),ad)) > r). When (d) is the result of (Dombrowski et al. 2019). first-order attack in the input space using PGD, we call this input attack robustness. Whereas if (f).adv is the result of a locally bounded version of the attack (Heo et al. 2019). first-order attack in the parameter space using PGD, we call this model attack robusiness. Intuitively, this measures the proportion of inputs that are robust to various attack methods.
Certified robustness: Using our certification method, we compute ((), according to Section 4.1 and then measure: 1/j Eiz (h(v"r, of (0), > T). Intuitively, this metric is the proportion of inputs which are certified to be robust. Where (.),,cert is the result of propagating input set T this input certified robustness, and when propagating M we call it model certified robustmess.
6.1 CERTIFYING EXPLANATIONS IN FINANCIAL DATASETS
In financial applications, it is of interest that users not be able to game financial models into provid- ing incorrect explanations by slightly changing their features. We study the robustness of NNs in financial applications to adversarial users attempting to game the NN into providing an explanation that shows the model is predicting on the basis of protected attributes (e.g., gender, race, age). In order to do so we study the Adult dataset, predicting whether or not an individual makes more or less than fifty thousand dollars a year, and the German Credit dataset, predicting whether an individual has a good or bad credit rating. Importantly. financial modellers should rigorously ensure/certify that their models are indeed not biased (Benussi et al.); we discuss this further in Appendix D We assess the robustness against gaming by taking enLh to be a vector with one entries for each of the sensitive features and zeros elsewhere, and we say an attack is successful if the adversary is able to force the sensitive feature (gender) into being in the top 5 explaining features by magnitude. That is, an attack is successful if the explanation reveals that the sensitive feature is one of the most relied upon features. In the top row of Figure 2 we present the accuracy and robustness values for standard training, gradient certified training (ours), and other robust explamation baselines including those suggested in (Dombrowski et al. 2022), Hessian regularization + softplus activation functions, and (Drucker & Le Cun 1992). training to minimize l gradient difference in a ball around the inputs. Objectives, further baselines, and details can be found in Appendix G On the Adult dataset find
7

---

Published as a conference paper at ICLR 2023
1321 150
19
6.6
40
62
6000
K3. 3EN DPV (E1 E101 615 R1%
275
Figure 3: Gradient certified training leads neural networks with considerably heightened robust- ness against targeted (left column) and untargeted attacks (center column). In addition, we find that our training leads to certifiably robust predictions (right column). Left column: Targeted attack robustness, higher is better, against adversary who tries to force explanation into the corner of an image. We see that FCNs are robust to this attack over the entire domain (e = 1.0). Center column: On the y-axis we plot the the average value of á normalized by the gradient magnitude, lower is bet- ter. We plot a red dashed line at the success threshold for untargeted attacks and find that FCNs are considerably more robust to untargeted attacks. Right column: On the y-axis plot the certified robust accuracy, showing that gradient certified training leads to certified prediction robustness.
that networks trained with (Drucker & Le Cun 1992) regularization and without any regularization (standard) are vulnerable to explanation attacks. On the German dataset, we find that overall, models are more robust to being fooled. However, we consistently find that the best robustness performance and the only certified behavior comes from gradient certified training.
Bias detection study We study the ability of explanations to flag bias in models. We create biased models by performing label poisoning. We set random proportion p of individuals from the major- ity class to have a positive classification and the same proportion of individuals from the minority class to have negative classification. The proportion p labeled 'induced bias" in Figure A is strongly correlated with the actual bias of the model, see Appendix D We then use the input gradi- ent, v, indicate if the explanation detects the bias in the learned model. This is done by measuring the proportion |1yl/ Ex= |eyl where j is the index of the sensitive feature this proportion is called "explanation detected bias' in our plots. We highlight that our method significantly out-performs other methods at detecting bias in the model likely due to the certified local linearity providing more calibrated explanations. While our method is promising for bias detection it should supplement rather than replace a thorough fairness evaluation Barocaa & Selbst 2016).
6.2 CERTIFYING EXPLANATIONS IN DIGIT RECOGNITION
In this section, we study the affect of input peturbations the explanations for networks trained on the MNIST digit recognition dataset. We train two networks, a two-layer fully-connected neural network with 128 neurons per layer and four layer convolutional neural network inspired by the architecture in (Gowal et al. 2018). We take a set of twenty target explanations to be binary masks with five by five grid of ones the corners with varying distances from the border of the image and based on visual inspection set r = 0.04, see Figure S in the Appendix for illustration. In Figure B we plot the result of our robustness analysis. We note that there is a consistent 2% test-set accuracy penalty for every 0.01 increase in G. We find that for both FCNs and CNNs we are able to get strong certification performance. We highlight that for FCNs we are able to certify that there is no input in the entire domain that puts its explanation in any corner of the input image, this is indicated by e == 1.0, as the images are normalized in the range [0, 1]. In the center column of Figure B we plot the results of untargeted attack analysis. In particular we take the untargeted e described in Section 4.1 and measure ||-- | ||o|||2 Again based on visual inspection, we set the untargeted
threshold 2.0, indicated with red dashed line. Finally, in the right-most column of Figure
8

---

Published as a conference paper at ICLR 2023
Pert
MILO Met ARRCCREA
MCc 06T.
0
8
Acoin
AP-0 let Mod. M830 Fob
19h Con
NeA. KM
Stansardd Trwining
Grad Cart. Towning
Standard
Gradi
On MedMNIST datasets the gradient certified loss improves attack robustness and gets strong input and model certified robustness. In addition, we find that explanations from a variety of methods are more sparse and intuitive than explanations from normally trained networks. Top rows: On DermaMNIST, the gradient certified loss leads to considerable increases in all robustness metries for only 5% reduction test-set accuracy. Bottom rows: On PneumoniaMNIST despite standard training having strong attack robustness performance only gradient certified training has non-trival certified robustness with only a 3% reduction in test-set accuracy.
Figure 4:
we plot how our training method affects the certifiable prediction robustness. Using bounds from (Gowal et al. 2018) we can certify that no input causes the prediction to change. We show that for both FCNs and CNNs our bounds lead to considerable, non-trivial certified prediction robustness.
6.3 CERTIFYING EXPLANATIONS IN MEDICAL IMAGE CLASSIFICATION
We consider two datasets from the McdMNIST benchmark and third in Appendix The first task, DermaMNIST, is to classify images into one of 11 different skin conditions. The second task, PneumoniaNNIST, is to classify chest x-rays into positive or negative diagnosis for pneumonia. Der- maMNIST is represented by full-color images (2352 feature dimensions) while PneumoniaMNIST represented by black and white images (784 feature dimensions). We use the same 20 binary cor- ner masks as in the MNIST analysis. We train models with
0.01 and 7/- 0.01 and test with e = 0.025 and ạ = 0.025. Across each dataset, we find that the only non-trivial certification comes from training with impressive attack robustness, we are unable to certify explanation robustness for standard trained models. This result mirrors the fact that only certified training yields non-vacuous prediction robust- ness bounds for high- high-dimensional inputs in the adversarial attack literature (Gowal et al.) 2018). We highlight that our method indicates that no model attack can be successful if perturbs the weights by 2.5% of their value, but we cannot make guarantees outside of that range, and in principle there will always exist classifier outside of the guaranteed range that corrupts the input gradient. In the left half of Figure M we plot example explanations from NNs trained with standard training as well as our training method. To avoid cherry-picking bias we use the first test image for each dataset. We find that the explanations provided by networks trained with our method are more sparse and seem to be better correlated with intuitively important features the input image.
method. Even for PneumoniaMNIST where the standard network shows
CONCLUSION
We present a computational framework for upper-bounding the largest change an adversary can make to the input gradient of neural network given bounded modification of the input features or model parameters. Our framework allows us to present, to our knowledge, the first formal certifi- cation of explanation robustness for general neural networks. We empirically validate our findings over six datasets of varying complexity and find that our method able to produce certificates of explanation robustness for each.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
MW acknowledges support from Accenture. MW and AW acknowledge support from EPSRC grant EP/V056883/1. AW acknowledges support from Turing AI Fellowship under grant EP/V025279/1, and the Leverhulme Trust via CFI.
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In NeurlPS Workshops, 2018.
Julius Adebayo, Michael Muelly, Ilaria Liccardi, and Been Kim. Debugging tests for model expla- nations. In NeurIPS, 2020.
Christopher Anders, Plamen Pasliev, Ann-Kathrin Dombrowski, Klaus-Robert Müller, and Pan Kessel. Fairwashing explanations with off mffifoliflld detergent. In International Conference on Machine Learning. pp. 314-323. PMLR, 2020.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of curity: Circumventing defenses to adversarial examples. In International conference on machine learning, pp. 274-283. PMLR, 2018.
Solon Barocas and Andrew Selbst. Big data's disparate impact. Calif. Rev., 104:671, 2016.
Elias Benussi, Andrea Patane', Matthew Wicker, Luca Laurenti, and Marta Kwiatkowska. Individual fairness guarantees for neural networks. Proceedings of the Thirty- International Joint Conference on Artificial Intelligence, IJCAI-22, pp. 651-658. doi: 10.24963/ijcai.20222 2022/992.
Leonard Berrada, Sumanth Dathathri, Krishnamurthy Dvijotham, Robert Stanforth, Rudy R Bunel, Jonathan Uesato, Sven Gowal, and M Pawan Kumar. Make sure you're unsure: A framework for verifying probabilistic specifications. Advances Neural Information Processing Systems, 34: 11136-111 11147, 2021.
Umang Bhatt, McKane Andrus, Alice Xiang, and Adrian Weller. Machine learning explainability for external stakeholders. In ICML Workshop on Human Interpretability, 2020a.
Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José MF Moura, and Peter Eckersley. Explainable machine learning in deployment. In Proceedings of the 2020 conference on fairness, accountabiliry, and transparency, pp. 648-657, 2020b.
Reuben Binns. On the apparent conflict between individual and group fairness. In Proceedings of the 2020 conference on fairness, accountability, and transparency, pp. 514-524, 2020.
Guy Blanc, Jane Lange, and Li-Yang Tan. Provably efficient, succinet, and precise explanations. Advances in Neural Information Processing Systems, 34:6129 6141, 2021.
Guy Blanc, Caleb Koch, Jane Lange, and Li-Yang Tan. A query-optimal algorithm for finding counterfactuals. In International Conference on Machine Learning, pp. 2075-2090. PMLR, 2022.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017
IEEE symposium security and privacy (sp), pp. 39-57. IEEE, 2017.
Jiefeng Chen, Xi Wu, Vaibhav Rastogi, Yingyu Liang, and Somesh Jha. Robust attribution regular- ization. Advances in Neural Information Processing Systems, 32, 2019.
Botty Dimanov, Umang Bhatt, Mateja Jamnik, and Adrian Weller. You shouldn't 't trust Learning models which conceal unfairness from multiple explanation methods. In ECAI, 2020.
Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus- Robert Müller, and Pan Kessel. Explanations can manipulated and geometry to blame. Advances in Neural Information Processing Systems, 32, 2019.
10

---

Published as a conference paper at ICLR 2023
Ann-Kathrin Dombrowski, Christopher J Anders, Klaus-Robert Müller, and Pan Kessel. Towards robust explanations for deep neural networks. Pattern Recognition, 121:108194, 2022.
Harris Drucker and Yann Le Cun. Improving generalization performance using double backpropa- gation. IEEE transactions neural networks, 3(6):991-997) 1992.
Mahyar Fazlyab, Manfred Morari, and George J Pappas. Safety verification and robustness anal- ysis of neural networks via quadratic constraints and semidefinite programming. arXiv preprint (9Xxvv//03.0// 287, 2019.
Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen Petar Tsankov, Swarat Chaudhuri, and Mar- tin Vechev. Ai2: Safety and robustness certification neural networks with abstract interpreta- tion. In 2018 IEEE S&P, pp. 3-18. IEEF, 2018.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In AAAI, 2019.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Bryce Goodman and Seth Flaxman. European Union regulations algorithmic decision-making
and a "right to explanation". Al magazine, 38(3):50-57, 2017.
Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Ue-
sato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arXiv:1810.12715. 2018.
Gunning. Explainable artificial intelligence (xai) darpa-baa-16-53, Defense Advanced Research Projects Agency, 2016.
Warren He, James Wei, Xinyun Chen, Nicholas Carlini, and Dawn Song. Adversarial example defense: Ensembles of weak defenses are not strong. l 11th USENIX workshop on offensive technologies (WOOT 17), 2017.
Juyeon Heo, Sunghwan Joo, and Taesup Moon. Fooling neural network interpretations via adver- sarial model manipulation. In NeurIPS, 2019.
Alexey Ignatiev, Nina Narodytska, and Joao Marques-Silva. Abduction-basse explanations for ma- chine learning models. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 33, pp. 1511-, 2019.
Adam Ivankay, Ivan Girardi, Chiara Marchiori, and Pascal Frossard. Far: general framework for attributional robustness. arXiv preprint arXiv:2010.07333,, 2020.
Junqi Jiang, Francesco Leofante, Antonio Rago, and Francesca Toni. Formalising the robustness of counterfactual explanations for neural networks. AAAI Conference on Artificial Intelligence, 2023.
Emanuele La Malfa, Rhiannon Michelmore, Agnieszka M. Zbrzezny, Nicola Paoletti, and Marta Kwiatkowska. On guaranteed optimal robust explanations for nlp models. In Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI-21, 2021. doi: 10. 24963/ijcai.2021/33
Himabindu Lakkaraju and Osbert Bastani. "How do fool you?" manipulating user trust via mis- leading black box explanations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp. 79-85, 2020.
Himabindu Lakkaraju, Nino Arsov, and Osbert Bastani. Robust and stable black box explanations. In International Conference on Machine Learning, pp. 5628-5638. PMLR, 2020.
Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of inter- pretability is both important and slippery. Queue, 16(3):31-57, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. In ICLR, 2018.

---

Published as a conference paper at ICLR 2023
Matthew Mirman, Timon Gehr, and Martin Vechev. Differentiable abstract interpretation for prov- ably robust neural networks. In International Conference on Machine Learning, pp. 3578-3586. PMLR. 2018.
Kiarash Mohammadi, for nearest counterfactual explanations. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 177-187, 2021.
Karimi, Gilles Barthe, and Isabel Valera. Scaling guarantees
Fábio Pinto, Marco OP Sampaio, and Pedro Bizarro. Automatic model monitoring for data streams. arXiv preprint arXiv:1908.0422: 2019.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High-precision model-agnostic explanations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Laura Rieger and Lars Kai Hansen. A simple defense against adversarial attacks on heatmap expla- nations. arXiv preprint arXiv:2007.0638// 2020.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206- 2019.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp- 618 128-626, 2017.
Nianwen Si, Heyu Chang, and Yichen Li. A simple and effective method to defend against saliency map attack. In International Conference Frontiers of Electronics, Information and Computa- tion Technologies, pp. 1-8, 2021.
Mayank Singh, Nupur Kumari, Puneet Mangla, Abhishek Sinha, Vineeth N Balasubramanian, and Balaji Krishnamurthy. Attributional robustness training using input-gradient spatial alignment. In European Conference on Computer Vision, pp. 515-533, Springer, 2020.
Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In AAAI, 2020.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint
2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328. PMLR, 2017.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties neural networks. arXiv e-prints, art. arXiv:1312.6199, December 2013.
Vincent Tjeng, Kai Xiao, and Russ Tedrake. Evaluating robustness of neural networks with mixed integer programming. arXiv preprint 1rXiv:1711.07335 2017.
Zifan Wang, Haofan Wang, Shakul Ramkumar, Matt Fredrikson, Piotr Mardziel, and Anupam Datta. Smoothed geometry for robust attribution. In NeurIPS Workshops, 2020.
Tsui- Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Duane Boning, Inderjit Dhillon, and Luca Daniel. Towards fast computation of certified robustness for relu networks. arXiv preprint arXiv: 2018.
Matthew Wicker. Adversarial robustness of Bayesian neural networks. PhD thesis, University of Oxford, 2021.
Matthew Wicker, Xiaowei Huang, and Marta Kwiatkowska. Feature- Feature-guided black-box safety testing of deep neural networks. In International Conference on Tools and Algorithms for the Construc- fion and Analysis of Systems, pp. 408 426. Springer, 2018
12

---

Published as a conference paper at ICLR 2023
Matthew Wicker, Luca Laurenti, Andrea Patane, and Marta Kwiatkowska. Probabilistic safety for bayesian neural networks. In Conference on Uncertainty in Artificial Intelligence, pp. 1198-1207.
PMLR, 2020.
Matthew Wicker, Luca Laurenti, Andrea Patane, Zhuotong Chen, Zheng Zhang, and Marta Kwiatkowska. Bayesian inference with certifiable adversarial robustness. In International Con- ference on Artificial Intelligence and Statistics, pp. 2431-2433 PMLR, 2021.
Min Wu, Matthew Wicker, Wenjie Ruan, Xiaowei Huang, and Marta Kwiatkowska. A game-based approximate verification of deep neural networks with provable guarantees. Theoretical Computer Science, 807:298-329 2020.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2921-2929, 2016.
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint 170v:1700..0455, 2017.
13


Reviews:
[{"review_id": "ugDvJUTtVZ", "reviewer": "Reviewer_wHym", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes certification methods for assessing input-gradient robustness to both perturbations in inputs and weights, and this is achieved using interval bound propagation (IBP) methods. To this end, the paper proposes to use box-type constraints to measure input-gradient robustness (definitions 1 and 2). The main methodological contributions are the usage of Lemma 1 on neural network, which presents a way to compute bounds of products of matrices; and section 5.1, which presents a regularized loss function based on the certified loss. \n\nExperiments show that the proposed certified loss function results in models that have improved gradient attack robustness across datasets, and sparser gradients with minimal degredation of test performance.\n", "summary_of_the_review": "Overall, this paper misses some crucial details about its central result that impact its novelty, misses relevant baselines, and conducts only small scale experiments. As a result, I am currently leaning towards a reject. \n", "strength_and_weaknesses": "**Strengths**:\n+ The problem considered is an important one, and the proposed direction of having certifiable bounds on gradients robustness has not been explored much in literature, and thus is novel and \ninteresting from that perspective. \n\n+ The proposed solution of presenting a common method that provides robustness both to input and weights perturbation via box constraints is novel as far as I am aware.\n\n**Weaknesses**:\n\n**Lemma 1: unclear novelty, missing proof, unclear notations, unclear slackness**\n\n- The central result for this paper (Lemma 1) seems to be a previously known result, and the contribution of this paper is its usage for forward and backward propogating interval bounds in NNs. However, this point is not stated clearly in the paper, as the introduction claims that \"We derive explicit bounds relying on interval bound propagation\" suggesting that they indeed derive Lemma 1. Perhaps they mean that they present a computational framework to **compute** bounds rather than derive them? It would be helpful to more clearly state the contributions of this paper in this case.\n\n- The paper states that \"We provide more detailed exposition and proof-sketch in Appendix X\", which I could not find in the supplementary material. Presumably these are proofs for Lemma 1 and their usage for deriving NN forward and backward propagation?\n\n- (Minor) The statement of Lemma 1 includes the notation M^B = A | B^r |. Just to clarify, is this a matrix multiplication of A with element-wise absolute value of B^r? It would be helpful to clarify this notation.\n\n- (Minor) The paper mentions that this Lemma may not provides the tightest bounds. It would be helpful to provide some discussion or intuition about how loose these bounds are, and under what conditions are they tight.\n\n- (Minor) I could not find Lemma 1 in the PhD thesis linked, so it would help to provide more detailed pointers regarding chapter number, etc.\n\n**(Minor) Robustness definition may not be practically relevant** \n\n- Definitions 1 and 2 present defns for explanation robustness in terms of box constraints. However, the **direction** of an explanation vector is more important than its magnitude which reveals the relative importance of various input features. Thus according to the current definition, assume there exists explanation vectors whose directions remains constant within inputs set \"T\" or parameter set \"M\", but only their magnitude changes. Then such models have non-robust gradients according to the present definition, but that may not be of practical significance as far as explanations are concerned.\n\n**Missing baselines: gradient regularization, local linearization and curvature regularization** \n\n- A missing baseline is gradient norm regularization, which penalizes the magnitude of the gradient explanation. The proposed regularizer penalizes both gradient magnitude and directions (see previous point), so it is important to disentangle the effects of magnitude vs direction regularization. \n\n- Another set of missing baselines are a local linearity regularizer (https://arxiv.org/abs/1907.02610), and curvature regularization (https://arxiv.org/abs/1811.09716) which explicitly penalizes model curvature, which in the limit forces models to be linear similar to the proposed method.\n\n- An alternative to these approaches is to compare against an adversarial training baseline, as all these methods are intended to be proxies for the same.\n\n**Missing evaluations on larger datasets and models**\n\n- Evaluations presently seem to be conducted on relatively smaller datasets (those that are MNIST-scale and below) and small models (two hidden layer neural networks). Some evaluations on larger datasets such as CIFAR / SVHN and larger models like deep resnets would help assess the scalability of the approach. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qlDzz-cMkx", "reviewer": "Reviewer_MkfV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a framework for assessing the explanation robustness of neural network (NN) models in the context of both classification and regression tasks. This proposed framework is claimed to solve a number of issues pertaining to the recent heuristic explanation approaches that suffer from a wide range of relevant adversarial attacks, i.e. \"out-of-distribution\" attacks, among many others. In particular, the paper develops a method to upper-bound the largest change an adversary can make in order to fool the corresponding explainer. The framework also allows one to train an NN model that is guaranteed to be certified explanation robust subject to a given parameter delta. Experimental results obtained for a few datasets demonstrate the efficiency of the proposed framework.", "summary_of_the_review": "Although the paper has a few issues outlined above, I am inclined to believe it makes an interesting contribution, which can be deemed sufficient for publication at ICLR.", "strength_and_weaknesses": "First of all, I should admit that I am no expert on heuristic explanation methods in machine learning so I might have overlooked some of the technicalities of the paper. However, I did my best to assess its general idea and its merits.\n\nI will start by listing the pros of the paper. I believe the paper is nicely written. The discussion is concise but informative enough for an average reader (like me) to follow. It clearly poses the problem with the existing explanation approaches and proposes a solution to the problem. Although I have not checked the proofs of the theoretical results, they make perfect sense to me. I should also say that experimental results look convincing to me.\n\nOn the negative side and despite my previous comments on the clarity, the paper mixes up the concept of explanations with the properties of the models themselves, which I find somewhat confusing. This can be observed in Definitions 1 and 2, which seemingly discuss the properties of an explanation while, in fact, they describe the properties of a model.\n\nHere I should also say that despite the claims made on explanation robustness, the authors fail to define what kind of explanations their methodology actually supports. Indeed, there are numerous post-hoc explanation approaches and the explanations they produce may vary a lot in terms of both syntax and semantics. It would be great if the authors could elaborate on this in their rebuttal.\n\nFinally, the authors overlook a large body of work on formal explainable AI, where abductive and contrastive explanations computed by means of formally reasoning about the target model are guaranteed to be sound (correct), i.e. no robustness issues apply to them. Furthermore, there are probabilistic extensions of the above, which are not discussed in the paper either. I believe these lines of work should have been discussed in the related work. I would also appreciate it if the authors could comment on this in the rebuttal.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LZ1ZMkU4_1", "reviewer": "Reviewer_tiM5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides certification guarantees on robustness of gradient based explainers by providing upper bounds on the largest adversarial change that can be made to the explanation from these explainers given bounded manipulation of either the input features or model parameters. Additionally, these bounds are differentiable, which means they can be used during training of neural networks that the explainers intend to explain and the resulting explanations will be provably robust. ", "summary_of_the_review": "I believe this paper is worth appearing in ICLR as it enhances our understanding of explainer robustness, and provides actionable tools for certifiably achieving it at training time.", "strength_and_weaknesses": "Strengths:\n- Important and significant research problem. Robustness of of explanation is an important problem from interpretability as well as fairness point of views. As authors point out, most current explainers are not very robust.\n- Theoretically grounded with provable certifications.\n- Actionable result as the results can directly be used to improve NN training from robust explanations point of view.\n- Thorough experiments on a variety of different datasets. \n\nWeaknesses:\nThese are minor points but can help\n- the $\\tau$ notation is overloaded between being a similarity threshold and a dissimilarity threshold. I don't think that confusion is needed either. Authors can stick with $\\tau$ as similarity threshold and simply flip the inequalities where it's currently being used as a dissimilarity threshold, this will also improve readability.\n\n- A few citations can enhance the intro further in the discussion of explainer robustness e.g. Alvarez-Melis & Jaakkola (2018) \"On the robustness of interpretability methods\" is a good reference discussing why robustness is important in explainers and how to quantify it. Khan et al. (2022) \"Analyzing the effects of classifier Lipschitzness on explainers\" and Wang et al. (2022) \" \"Robust Models Are More Interpretable Because Attributions Look Normal\" \" discusses the connection between smoothness of classifiers and robustness of explanation. Zhou et al (2022) \"From local explanations to model understanding\" and Ju et al (2022) \"Logic traps in evaluating attribution scores\" provide some useful counterpoints in the discussion about the importance of robustness for explainers.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2_MhQjWVc4", "reviewer": "Reviewer_MnSc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a bound on how much a gradient based explanation can be manipulated (by input OR model parameter manipulation). They then include the bounds into the training regime to create certifiably robust networks. They compare their approach to previous attempts to make neural network explanations more robust on several data sets. The thorough quantitative and qualitative evaluation shows the advantages of the authors' proposed methods.", "summary_of_the_review": "Overall I think this is a decent contribution. The authors could improve their work by providing a more thorough theoretical analysis. They might also consider lowering their claims and discussing the limitations of their approach critically. I'm happy to change my recommendation when my questions are answered to a satisfactory degree.", "strength_and_weaknesses": "### Strengths\n\n- important topic\n- theoretical justification/motivation\n- comparison to previous works\n- quantitative and qualitative evaluation looks very promising\n- the authors address robustness to input AND model parameter manipulation\n\n\n### Weaknesses\n\n- a more thorough theoretical analysis of differences to previous work would have been interesting\n- no evaluation on high dimensional image data (ImageNet)\n- the authors claim a bit too much for very little theoretical justification. After all they are still just solving an optimization problem, so there are not really guarantees.\n\n### Question and Remarks\n- I don't quite understand why in section 3 you define the gradient explanation as the gradient of the LOSS of the neural network output. It is usually just defined as the gradient wrt the winning class neuron of the network... The loss $\\mathcal{L}$ is also not defined anywhere. Is that the training loss? Why?\n- What is $J$ in the Backward Pass w.r.t. Bounds equations on page 6?\n- Where is the proof sketch and where is Appendix X?\n- Why can your certification procedure from section 4.1 not also be applied to standard, L2 Regr and Hessian Regr approaches?\n- You talk a lot about guarantees and provable robustness. But the theory does not provide these guarantees and empirically you would have to test all possible perturbation configurations in your defined ranges which you probably didn't do. \n\n### Typos/ minor remarks\n- in 4: Now, we proceed define what it $\\rightarrow$ Now, we proceed to define what it\n- in 5: As desired, Lemma 1 allows us to jointy propagate $\\rightarrow$ As desired, Lemma 1 allows us to jointly propagate\n- choosing $\\mathcal{L}$ for loss and loser bound seems a bit unfortunate.\n- in 5.1: For input robustness, implies that for all points in $\\rightarrow$ For input robustness, this implies that for all points in", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster