Ablation Path Saliency
Authors: Olivier Verdier
Justus Sagemüller
Venue:  (2021)
Keywords: image classification
interpretability
feature attribution
saliency
ablation

Abstract: We consider the saliency problem for black-box classification. In image classification, this means highlighting the part of the image that is most relevant for the current decision.
We cast the saliency problem as finding an optimal ablation path between two images. An ablation path consists of a sequence of ever smaller masks, joining the current image to a reference image in another decision region. The optimal path will stay as long as possible in the current decision region. This approach extends the ablation tests in [Sturmfels et al. (2020)]. The gradient of the corresponding objective function is closely related to the integrated gradient method [Sundararajan et al. (2017)]. In the saturated case (when the classifier outputs a binary value) our method would reduce to the meaningful perturbation approach [Fong & Vedaldi (2017)], since crossing the decision boundary as late as
possible would then be equivalent to finding the smallest possible mask lying on
the decision boundary.
Our interpretation provides geometric understanding of existing saliency methods, and suggests a novel approach based on ablation path optimisation.

Under review as a conference paper at ICLR 2021
Ablation Path Saliency
Anonymous authors Paper under double-blind review
Abstract
We consider the saliency problem for black-box classification. In image classi- fication, this means highlighting the part the image that is most relevant for the current decision. We cast the saliency problem finding tion path between two images. An ablation path consists of a sequence of ever smaller masks, joining the current image reference image in another decision region. The optimal path will stay as long as possible in the current decision region. This approach extends the ablation tests Sturmfels al. (2020). The gradient of the corresponding objective function is closely related to the integrated gradi- ent method Sundararajan al. (2017). In the saturated case (when the classifier outputs a binary value) our method would reduce to the meaningful perturbation approach Fong & Vedaldi (2017), since crossing the decision boundary as late as possible would then be equivalent to finding the smallest possible mask lying on the decision boundary. Our interpretation provides geometric understanding existing saliency methods, and suggests novel approach based ablation path optimisation.
optimal abla-
Introduction
The basic idea of saliency attribution is to provide something from which a human can judge how a classifier arrived its decision of the prediction gives for certain input. is difficult give a more mathematical definition, but various properties that such method should fulfill have been proposed.
Sundararajan et al. (2017) give axioms, of which sensitivity comes closest to the notion of saliency. Essentially, the features on which the output is most sensitive should be given higher saliency value. The authors give further axioms to narrow it down implementation invariance, completeness, lin- earity and symmetry-prestt preservation and obtain a corresponding method: the integrated gradient method. Note that have another way to arrive similar method, see $4.1
Fong & Vedaldi (2017) is closer to our work: the authors directly compute the saliency of given pixel by deleting, or altering that pixel, to see how this affects the output of the classifier.
Our method is to define a proper maximisation problem as follows. First, we define ablation paths time dependent smooth masks yp: [0, - C(S2, RR) going full mask empty mask, such that at each pixel the mask value decreases over time (see Figure 1).. We also impose constant area speed:
Figure I: Example of how an ablation path (sequence of masks, top) gives rise to transition between current target (image of house finch) and baseline (orca whale).

---

Under review as a conference paper at ICLR 2021
Let F be the classifier, that outputs a probability between zero and one. We choose a current image of interest 2o and a baseline
the area covered by the mask should increase linearly over time (see 33)
$ $4). Assuming that F( = 1 and F(x1) = O, maximising the objective function means that we try to find an
image 21. The objective function is then P(() fo F (t) zo)) df (see
ablation path that stays as long as possible in the decision region of xg- Intuitively, we try to replace
as many pixels of zo by pixels of x1 while staying in the same class as 20-
The main contribution of this paper is to formulate the saliency problem as an optimisation problem on paths. Doing so connects previous notions of saliency, notably integrated gradients Sundararajan et al. (2017) (see S 4. 1). the ablation tests Sturmfels et al. (2020) (see $ $4.2), 2), and meaningful pertur- bations Fong & Vedaldi (2017); Fong et al. (2019) (see & 4.3). Our formulation is also resolution invariant (it does not assume that images are defined on pixels); this allows to make clear differ- ence between images and their duals, for instance, which gives guidance to where regularisation
is needed.
2 Related Work
Simonyan et al. (2013) defines a saliency map as the gradient of the network output at the given image. This would appear to be a sensible defimition, but the resulting saliency is very noisy because the network output is roughly constant around any partieular image. SSllaraju et all (2016) improves the situation by computing the gradient after each layer instead. This is, however, not a black-box method such as the one we propose. Koh & Liang (2017) computes an influecc function, that is, a function that measures how the parameters would be changed by change in the training data. Although is a black-box method, it is not saliency method per se. They use the gradient of the network output to find the pixel most likely to have high saliency. The pixel that have most effect are given a higher saliency. By contrast, Petsiuk et al. (2018) proposes to directly evaluate the saliency by finding out which pixels most affect the output, similarly to Fong & Vedaldi 7). but without using any gradients.
There number of meta-studies of saliency methods. Adebayo et al. (2018) lists essential prop- erties, for instance the requirement that the results should depend on the training data in a sense that perturbing model parameters should change the saliency. Kindermans al. (2017) 7) proposes a num- ber of property that saliency methods should satisfy. Ancona et all (2017) compares several saliency methods and proposes method to evaluate them (the sensitiviy-n property).
3 Ablation Paths
3.1 Images and Masks
We consider a compact domain Ω. Note that Ω may be discrete or continuous: in fact, we assume that Ω is endowed with measure which could be the discrete measure (iff f. is set of pixels) the Lebesgue measure (if Ω is a domain in IR2, for instance). In the sequel, So denotes integration with respect to that measure. Without loss of generality, we assume the mass of that measure to be one, i.e., Jg 1.
We consider module M of functions on Ω with values in a vector space (the dimensions of V represent the channels, and elements of M represent images). This module is equipped with commutative ring which represents masks. Concretely, most of the paper we choose
M := C(2.V)
R := C(2, R).
The module structure simply means that masks can multiply images, i.e., that the operation Or gives a new image in M when R and x E M, and that this operation bilinear.
3.2 Ablation Paths
Definition 3.1. We define the set A of ablation paths as the set of functions :p: [0,1] + R. fulfilligg the following properties:
Boundary conditions (0)) 0 and (01))
2

---

Under review as a conference paper at ICLR 2021
Monotonicity 3 (((1) $ ptt2) Constant speed Sn (tt) =t t [0.
t, [0.1]
We will call monotone paths the paths that verify the first two conditions but not the third.
Note that the set of ablation paths is
subset of - ((0, 1], R)-
Some comments on each of those requirements are in order. (i) 0 and denote here the constant functions zero and one (which corresponds to the zero and one of the algebra It) (ii) (tt1) < ((t2) should be interpreted as usual as (222) (p(4) being in the cone of nonnegative elements (iii)
t + fn (t) is differentiable, this requirement can be rewritten as गi Sa +(()) = 1, so it can be regarded as constant speed requirement. This requirement more normalisation than require- ment, as is further detailed in Remark 3.3
There is canonical, simplest, ablation path between zn and 21; ((t) =
(1)
The mask is thus constant in space at each time t. The reader should check that all the requirements for an ablation path are fulfilled.
Note that an ablation path without the constant-speed property can always be transformed into one that does fulfil This is clear if the function + Jo pt) strictly increasing, this then just time reparameterisation, but this is in fact always possible, in canonical sense. The proof is in Appendix A.
Lemma 3.2. To any monotone path there corresponds canonical ablation path.
Since R is itself a function space, an ablation path yo is in fact a function two arguments. In the sequel, will abuse the notations and write yp function of one two arguments depending on the context: ep()) = (t,,-). -). For instance, in the definition Definition 3.1 above, Sa p(() = Sa p(t,,) = Sn (),,
Remark 3.3. the ablation path sp differentiable time, the requirements Definition addit remarkable reformulation. Define vb(1) := ((). All the requirements in Definition 3.1 are equivalent to the following requirements for function vi [0,1] Ω - RR
rt(, 2 O, J (sttrr) dr dr =
f 10.1 (t, =1 t [0.
The corresponding ablation path is then recovered by p(t) = that differentiable ablation paths can be parameterised densities of doubly stochastic Markov transition kernels on [0, x Ω.
fn (s) ds. What this means is
3.3 Regularity of Ablation Paths
Lemma 3.4. If ip is an ablation path, then
I1{p(tz) - --oolllttlllll = |t1 tol.
In particular, + (p(t.)) continuous as a function [0, 1] (1))
Proof. Choose to, in [0, 1]. Without loss of generality, assume t1 2 to. Then, fglee(t)) (p((t) pp(t)) t for from which conclude that yp(t1) (p(to) is in L and fulfiils the equation above.
4 Score of an Ablation Path
We now fix two points xo (the current image) and #1 (the baseline image) the space of images M. We propose the following measure of the score of an ablation path (see Definition 3.1) with respect
Here we can define the cone of nonnegative functions by { E C(S,R) 2 0). In general star algebra, this cone would he defined (I E R Fy R v v
3

---

Under review as a conference paper at ICLR 2021
to these two images. Given mask 0 E R, define the interpolated image (2o, æile M as 1 = (1 - f) + 0.1-
We now define the score function P: IR from ablation paths to R by the integral
P(e) = f.' F([ xile(t))dt.
Note that, F is bounded between zero and one, so is P(p) for any ablation path sp The main idea that if F(xo) and F( 0, the higher this value of P is, the better the path is to describe the salient parts of the image.
Note that the function is defined regardless of the constraints placed on ablation paths, i.e., the score function P is defined on the vector space of functions yp: [0. R. It is straightforward to compute its differential dP on that space:
(dP, 0④) - f. (dl (df
(41 Fo) 6sp(()) dt.
EM'
EM
ER
So if we define the product of E M and x M producing an element in He by (xD,4) =
(D, zxp)) as is customary, we can rewrite this differential as
(dP, 8e) - -f, ((x) ((11- xo)dF' mo 8ip(t) dt.
Note that we know that any ablation path is bounded, y ( 1]. R), the differential of P at can be identified with the function dP + (x1 zo (F) (lo in (1). R*).
4.1 Relation with the Integrated Gradient Method
When this differential is computed on the interpolation path & 11 and then averaged, then this is exactly the integrated average gradient Sundararajan al. (2017) More precisely, the integrated gradient is exactly fo d d/. Note that this in fact an integrated differential, since we obtain an element in the dual space M and this differential should be appropriately smoothed along the lines 5..1
4.2 Relation to Pixel Ablation
Given a saliency function E R and define (0) == 0, ((1) == 1. This path is monotone path, except in the module of images M = L (S, V), equipped with the ring of masks R = - (22). To an ablation path, it still needs to be transformed into constant speed path, which is always possible as explained in Appendix A.
define path by 5(() 1 slo(((((1-)) when / E (0.
Note that this is generalisation of the ablation method in Sturmfels et all (2020). In that case, the set Ω would be discrete set of pixels. Note that in the ranking, pixels with the same saliency would be ranked an arbitrary way and added to the mask in that arbitrary order. the method above, add them all at once, but the time reparameterisation keeps that function constant longer for however many pixels were ranked the same. As long as the ranking is strict (no two pixels have the same saliency), the two methods are the same.
4.3 Relation Meaningful Perturbations
In the saturated case, that is, if F only takes values zero and one (or in the limit where it does), our method basically reduces to finding the interpolation with the largest mask on the boundary, in essence the approach of Fong & Vedaldi (2017). Indeed, suppose that the ablation path yo crosses the boundary at time t" means that F F[22. Z1lptt)) has value one until t" and zero afterwards, so the score defined in ( is P(p) t* By the constant speed property, t* = Sq s(t"), so we end up maximising the mask
the boundary.
4

---

Under review as a conference paper at ICLR 2021
5 Optimisation Problem and Algorithm
We proceed define the optimisation problem at hand and how to solve numerically. Conceptually we try to find the ablation path (see Definition 3.1) that maximises the score P(p):
max P()).
PEd
Recall that the set A of ablation paths is convex; however, since the objective function P is not convex, this is not convex optimisation problem.
The method we suggest to follow gradient direction. Such an approach is in general not guaran- teed to approximate global maximum, but empirically it does manage that quite well here.
5.1 Gradient and Metric
Note that the differential is an element of (1. L (Io, 11, R). For now we assume that E <3, R) and dP E
we need a map from that space to
However, we still need a covariance operator K: R - R. In practice, we use a covariance operator associated to a smoothing operator. For measure μ E R², (K 0) == (M. Sq k(- ((-rr)((r) dr), where k is a suitable smoothing function, for instance k(r) cxp(-|1rl|//o) This allows us to consider the gradient of P. Note that different choices of metric will influence on the algorithm.
Since the optimisation problem constrained (since yp constrained by the requirements in Defini tion 3)),, following the gradient direction will lead us to violate the constraints. Since the constraints are convex, it straightforward enough to project each gradient-updated version back to something that does fulfill them, and indeed that is the idea behind our algorithm, however in practice it does by itself not yield convergence without impractically many iterations. See Appendix for the details of how we actually proceed.
Examples
To test out our path-scoring approach and the saliency method based on its optimisation, we use standard image classifier deep-CNN (Inception v4 Szegedy et al. (2016), pre-trained on ImageNet), with a selection of real-world images for both the current targer and haseline inputs. For each ex- ample pair, we compare multiple saliency methods.
Our algorithm yields whole path of masks, which cannot whole be visualised in one 2D view. Generally, the threshold mask, which we define as the mask right where the path crosses the decision boundary, is most insightful: it still preserves enough of the image to be classified correctly, but no more. All of the images in this section refer to that selection, and where the threshold lies in the path is indicated by the vertical black line in the score plots. To be precise, this is the mask that preserves as little of the image as possible whilst still resulting in the intended class keeping the highest probability among the classes in the classifier's prediction. Note that although the threshold mask encapsulates a large part of our method's ''s useful output, we find that the additional information from the rest of the path, and the score-plot along the path, also provide good diagnostics especially in examples where the mask highlights an unexpected region of the image.
Figure 2 is an example with particularly clear-cut interpretation: the vibrantly coloured rubber eraser is sufficient to retain the classification even in an almost completely ablated image. All of the com- pared methods manage to find a mask that focuses in on that, though with differences in the details. The unconstrained optimisation narrows it down to few individual pixels, which gives an extremely good score (somewhat unsurprisingly, since that what is being optimised), but the result is hardly useful for interpretability purposes: this should be considered an adversarial example. One interpre- tation of this is that identifying the gradient with the differential implies that the space of masks is essentially bounded functions wihout further regularity (see 15. similar to the mask space in 84.2 The region boundary in that space seems to be extremely close to the baseline, and the optimisation method finds those adversarial examples.
5

---

Under review as a conference paper at ICLR 2021
Optimised
1.0
0.5
score
0.0
0%
20%
40%
600
80%
100%
Optimised Path (filter regularisee))
1.0
0.5
score 0.955
0.0
Integrated Gra Gradients
0.858
0.0
100%
Filtered Integrated Gradients
0%
20%
40%
60%
80%
100%
Random Pixel
3.0
0.5
score 0.315
0.0
0%
20%
40%
60%
80%
100%
Ablation
Figure 2: Comparison of saliency methods for an image of pencil, against one of an orca whale as the baseline.
The integrated gradient method in principle also has this issue, and indeed the corresponding mask (pixel ranking, cf. 4..2 quite grainy/noisy, without however behaving adversarially (the entire pencil is highlighted).
The authors in Fong & Vedaldi (2017) were confronted with similar problems. We use smoothing filters to compute the gradient, in order to both avoid adversarial examples and to have less noisy saliency masks. Using that, the path-optimisation still manages achieve high score, but now highlights the eraser single, clearly interpretable feature.
Filtering can also be applied to integrated gradient before pixel-ranking. That does avoid the noisy- ness, but it also leads to blobby shape of the masks.
Comparison with a random-order ablation confirms that the good scores of the saliency methods really are result of non-trivial feature highlighting, rather than something that comes out of the transition between the images automatically 3 In Figure 3 is evident that the saliency methods do not in general agree so clearly. Here, the filtered optimal path again highlights small, single region that is sufficient for a classification of the image as a house- rather than goldfinch. Arguably, this is again adversarial (a human would certainly label the composite image gold- rather than house finch). However does give rise to useful interpretation: note that the highlighted region includes
'ee Appendix E for small, non-rigorous statistical study suggesting that scores > 0.9 are p 0.01 significant against null hypothesis of smoothly random masks, and extremely unlikely with pixelwise-random masks.

---

Under review as a conference paper at ICLR 2021
the house finch's 's plumage, whilst covering specifically the wing of the gold finch (which features a distinctive pattern not seen on house finches). So in this case, the saliency tells more about the baseline than about the current target.
The integrated gradient meanwhile hardly manages to mask out anything at all, before the classifi- cation switches to gold finch.
Practically speaking, saliency is particularly interesting in questions of whether a classification was actually correct. Figure 4 shows example where the model misclassified an apple as fig. The unstable scores in even the optimised ablation path are perhaps symptomatic of this (we are not sure how representative this is); nevertheless both our method and integrated gradients find mask of less than half of the image that still retains the fig-classification. Whilst with integrated gradients, this mask includes the apple (which might in a real-world unclear situation lead one think the classification payed attention to the intended target, increasing credibility), out method highlights mainly a patch of background at the border of the image.
The optimised paths depend not only on the model and current target, but also on the baseline; notice for example that the pencil, of which in Figure mostly the eraser was highlighted, is in Figure mostly characterised by the eraser's metallic joint as well as part of the blue shaft, which happens to coincide with the gold finch's 's wings that were also masked out in Figure 3 In Figure 6 it is something in between. Still the results (with filtering) tend to be relatively coherent across baselines, more so than with the Integrated Gradients or the adversarial unfiltered ones.
One might ask why to use a true image a baseline at all (as opposed to blurred image, constant colour image, as in Sturmfels et al. (2020) or Fong & Vedaldi (2017)). The problem with artificial baselines is that the network missclassifies them (a blurred goldfinch is classified as power drill, for example), so the ablation path crosses region where the network is extrapolating wildly. The resulting saliency may be difficult interpret since the -unknown- parts of the baseline which the networks considers as important (which part of the blurred goldfinch lead to the power drill classification?) have an influence on the saliency of the current image. Ideally, we would like have some result about saliency that gives good scores across many different baselines. Future research is needed.
Another choice be made is the regularisation. We used here Gaußian filters; different sizes com- pared in Figure C. It is a tradeoff between blurring out the boundaries and inviting noisiness, however even a small filter appears be enough avoid the algorithm from converging adversarial exam- ples (seemingly smaller than what Fong et al. (2019) require). It is even possible to scale down the filter during the optimisation to eventually give very sharp edges but not noise elsewhere, though is somewhat dubious what sense this makes mathematically, from the point of view that the filtering represents just metric on R. Again, further research is needed to assess the reliability.
Optimised Path (filter regularised)
0.5
0.938
0.0
Integrated Gra dients
0.5
0227
0.0
80%
100%5
Ablation
Figure 3: Method comparison for house finch image against gold finch baseline.

---

Under review as a conference paper at ICLR 2021
Optimised o=10)
1.0
0.5
score 0:60s
0%
20%
40%
60%
80%
100%
Integrated
1.0
score 0:00
0%
20%
40%
60%
80%
100%
Ablation
Figure 4: Method comparison for a misclassified image: Inception classifies the apple as fig in- stead.
Optimised Path (filter-reullarised)
1.0
score 0.041
0.0
0%
20%
40%
60%
80%
100%
Integrated Gra Gradients
1.0
score 0.68
0.0
0%
20%
60%
80%
100%
Figure 5: Comparison of saliency methods for an image of a pencil, against one of a gold finch as the baseline.
7 Pointing game
We evaluate our saliency algorithm using the pointing game method. This method was introduced in Zhang et al. (2017) and used, for instance, in Selvaraju et all (2016); Fong & Vedaldi (2017). The primary goal is to show that our method, applied to good enough image classifier, typically gives results that match what human would also think of as the important part of the image. Specifically, we check whether the maximum-salient pixel lies within the bounding box of the object that defines the class.
Table shows some results for our method on various images with blurred image as baseline. We show a few examples of this pointing game on Figure See Appendix D for details and caveats with these results.
Class (sample size)
Ablation path
Meaningful Perturbations
Bee (121)
83 %
46 %
Power drill (121)
77 %
40 %
Saxophone (119)
%
49
Table 1: The success rate of the pointing game for various classes from the ILSVRC2014 dataset Rus- sakovsky et al. (2015). We chose the first images in each class by alphabetical order. We compare to the meaningful perturbation method in Fong & Vedaldi (2017) (see caveat at the end of opppnddxx D)
8

---

Under review as a conference paper at ICLR 2021
Optimised
1.0
0.5
Score
0.0
0%
20%
40%
60%
80%
100%
Optimised Path (filter: c=
1.0
0.5
5core 0.926
0.0
60%
Optimised Path (filter: a=10)
0.953
0.0
60%
100%
Optimised Path (vanishing filter)
0%
20%
40%
60%
80%
100%
Integrated
3.0
0.5
W
score 0.544
0.0
0%
20%
40%
60%
80%
100%
Ablation
Figure 6: Comparison of saliency methods for an image of a pencil, against one of house finch as the baseline.
Figure 7: Examples how the ablation path saliency "points" in the bounding box of various images of becs. The bounding boxes are human defined in the datasct, the red crosses indicate the location of least-ablated pixel.
References
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances Neural Information Processing Systems 31. pp. 9505-9515. Curran Associates Inc.. 2018 URL http://papers: www.. cttpsperccc/////c/c/p 8160- 1e000-saniyy-ceccke--ee sar-ecccee------aaaeeenn--aaaa pdf.
Marco Ancona, Enca Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. CoRR, 2017.

---

Under review as a conference paper at ICLR 2021
Ruth Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. CoRR, 2017. URL attp:///rxixi wttp:/iww..rgg/b//////////// 03296v3.
Ruth Fong, Patrick Mandela, and Andrea Vedaldi. Understanding deep networks via extremal per- turbations and smooth masks. ICCV, 2019. URL http:///aaarra..or/////////00000 08485
Jacob Gildenblat. Pytorch
boxes meaning- 2017.
ful perturbation.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un)reliability of saliency methods. CoRR, 2017. URL
prediicctosooiiiiii Pang Wei Koh and Percy Liang. Understanning black predictions via influence functions. CoRR, 2017. URL
04730v2
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. CoRR, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-252, 2015, doi: 10.10/ 1203-01------11-1
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. CoRR, 2016. URL http://arxiv org/ abtp///bbb1......././ 02391v4.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, 2013.
Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Visualizing the impact of feature attribution base- lines. Distill, 2020. doi: 10.23911/distill.o0000 https///istilll.pub//000 2ttp/////rrbuuttoonaa/aaaaeeeeee
Mukund Sundararajan, Ankur Taly, and Oigi Yan. Axiomatic attribution for deep networks. CoRR, 2017. URL http://arxxtv.orr/abb///// 01365
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alex Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning, 2016.
Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. CoRR, abs/1905 1905.11946, 11946, 2019. URL
11946.
Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126(10):00441110, 2017.
doi: 10.1007/sl 1233-077-1059-
A Canonical Time Reparametrisation
Proof of Lemma 3.2. The function [0, 1| + IR defined by m(t) == So stt) is increasing and goes from zero to one (since assume that fn 1).
Note first that if m(t1) = m(t2), then (p(t)) (tz) from the monotonicity property. Indeed,
supposing for instance that & < t2, and defining the element = ((22) (t()) we see that on the one hand Sa 0 O, on the other hand, 0 > 0, so 0 0 and thus (e(t)) (12))
Now, define M == m 1)) = {s [0. 3t E [0, 10 = s }. Pick s E [0.
If M we define (s) == (tt) where m(t) s (and this does not depend on which fulfills
mt() = s from what we said above). We remark that Sa (bs)) = Sa ppf) = m(t) =s
Now suppose that & M. Define s1 = sup(M n [0, s)) and 82 = inf n [s, 1]) (neither set are empty since M and 1 E M). Since $1 M and 82 M, there are E [0, 1] and lo (0, such
10

---

Under review as a conference paper at ICLR 2021
that mtt1) $1 and m(t2) 82. Finally define (5s)) ((11) (s $1s ( ( In this case, Sa (s)) m(t1) (s $1) m((a)-m((x) The path & constructed this way is still monotone, 52-51
*2
and it has the constant speed property, so it is an ablation path.
B - -pptimall Monotonicity Projection
The algorithm proposed in Appendix C for optimising monotone paths uses updates that can locally introduce nonmonotonicity in the candidate & so that it is needed to project back onto a monotone path sp1. The following routine performs such a projection way that is optimal in the sense of minimising the _ -distance i...
sup (t, - (i(t,r)) sup (t, (t,rr)
for all Ω and any other monotone path v.
The algorithm works separately for each r, i.e. we express it as operating simply on continuous functions p [0, + R. The final step effectively flatlens out, in minimal way, any region in
Algorithm Make function [0, + nondecreasing
U:14,, ra] - { [0, p1( 50}
Union of intervals where p decreases
for i do
m; f- p(h))p((r)
4 - max & 4 p(t) < m, }
2
ri < min t E [rs, 4-
p(t) 2 m;
end for for j.j do
if [4,r::] r [g A 9 then
if m, my, merge the intervals and recompute m as the new center
end if
end for return f
if & (.4,, ((() m ift [.
which the function was decreasing.
In practice, this algorithm is executed not on continuous functions but on a PCM-discretised repre- sentation; this changes nothing about the algorithm except that instead as real numbers, l, and & are represented by integral indices.
C Path Optimisation Algorithm
As said in tsection S repeatedly seeks the direction within the space of all paths that (first ignoring the monotonicity con- straint) would affect the largest increase to P(y)) as per (2). As discussed before, this choice already requires a metric to obtain a vector-gradient from the covector-differential which could be either the implicit ( metric on the discretised representation (pixels), or more physical Bernel/filter-base metric. We conceptually use the latter, however for technical reasons do not immediately apply the corresponding filter to the differential but rather to the path, which is not quite equivalent but does have the effect of avoiding noise from creeping into the state.
our optimisation algorithm is essentially gradient descent of path it
Unlike with the monotonisation condition, the update can easily be made preserve speed-constness by construction, by projecting for each the gradient g on the sub-tangent-space of zero change to
It is easy to come up with other algorithms for monotonising (discretised) function. One could simply not optimal with respect to any of the usual function norms; or clip the derivatives to he nonnegative and then rescale the entire function, but that is not robust against noise pertubations. Note that the optimum not necessarily unique.
sort the array, but
11

---

Under review as a conference paper at ICLR 2021
0.5
(b) Decreasing intervals
022
0/6
(a) Original
0.5
(c) Grown intervals
0.5
(c) Monotone projection
(d) Merged intervals
Figure 8: Example view of the monotonisation algorithm in practice. contains decreasing inter- vals, which have been localised in (b). For each interval, the centerline then extended to meet the original path non-decreasingly (g) some cases, this will cause intervals overlapping; in this case merge them to single interval and re-grow from the corresponding centerline (L). Finally, replace
the path on the intervals with their centerline (g).
Algorithm 2 Projected Gradient Descent
1: ip ((t, r) + t)
Start with linear-interpolation path
2: while y0 is not sufficiently saturated do
3: for in [0, 1] do
4:
Fept (1 ((() xp + sp()
5:
compute F( with gradient = (Fip.))
6:
D ensure does not affect mass of (+()) (ğ(r) | za) for r in S2 (optional) apply filter yp(t)
letg = g --
7:
update cp(t, < s
D by is learning rate
8:
end for 10: (optional) apply nonlinear gain to yp
9:
11:
for in Ω do
12:
re-monotonise + (tt,, r), using Algorithm
13:
end for
14:
clamp (tt, r) to [o, 1] everywhere
15:
re-parametrise yo, such that Jn ptt) t for all (using Appendix A
16: end while.
Ja (t)), by subtracting the constant function times Sn g(t). Note this requires the measure Ω to be normalised, else considered this point.
Then we apply these gradients time-wise as updates to the path, using a scalar product in the channel- space obtain the best direction for yp itself (as opposed to the corresponding image composit x..)))
Simply projecting the updates path then again to the set of legal (in the sense of Definition 3.1) abla- tion paths would presumably enough to converge towards saturated path with high score, however
12

---

Under review as a conference paper at ICLR 2021
1-10-2
2-10-2
-2.10 10-2
-4.10-
0.2
0.4
0.6
0.8
Figure Function e: [0, 1] - R. Note that |e(x)] (2 - (12))==//))// + 1/2 - x, with Ç = 0.8.
5
10-2
The formula is e(x)
in tests with artificial constant gradient we found out that this requires extremely many iterations, with the number apparently scaling with the image dimension. The problem here is that saturating updates tend be undone again by the nored-normalising reparametrisation, except for the most affected pixel. (Just increasing the learning rate does not help with this.) If the gradients come from a deep CNN and for every pixel the monotonicity needs to restored, such many iterations would be prohibitly computation- -inteiatiiooinntesiveee.e..
Fortunately we can dramatically speed up the convergence by artificially encouraging saturation. We tweak the ablation path to pointwise with sigmoidal function that brings values lower than 2 slightly closer O, and values greater than slightly closer 1. To this end, a perturbation of the identity function defined by x + x + (x)), and apply pointwise to the path:
p(r, t) (pr,t) c(v((r,t)
The perturbation function e: [] IR has the property that x + e(x) maps [0. 1] into itself. The function we use plotted on Figure 9. Although this perturbation seems small, the examples in $ now only require 20-40 iterations.
As to the motivation behind the transformation y + 4p ap+e(pp) <(p), notice first that fully saturated masks - i.e. those that choose for every pixel are either zero or one, selecting exactly the value of either the current target or the baseline- - are fixpoints of the function r ++ 2 ee(x) since e(0) = <(1) = O. So if such mask is the optimum in the algorithm without artificial saturation (while there is no guarantee for this in general, this seems fairly frequent in practice), then will also be an optimum of the algorithm with artificial saturation.
What is more, the dynamical system x + (x() quickly converges to zero or one, which efficiently encourages saturation, without sacrificing precision, the function chose is quite small.
Conversely, and unlike high learning rate, the saturation function is by construction monotone, sym- metric and keeps the signal within the allowed bounds, it avoids violating the ablation path con- straints. The problem with high learning rates is that the algorithm gets caught in a sequence of alternating strong constraint-violating updates followed projection step that largely undoes the previous update.
D Evaluation with Pointing Game
The Pointing Game, the results of which show in is way verify that the saliency method points at a region that human would consider relevant to the classification. It is often the region of image that contains the physical object which is being classified.
13

---

Under review as a conference paper at ICLR 2021
Such evaluations of saliency methods clearly have caveats. One can for instance argue that the cases when the saliency points somewhere outside the bounding box are the most insightful ones, as they indicate that the classifier is using information from unexpected part the image (for instance, the background). Another caveat is that, if winning at the pointing game is the goal, a saliency method is only as good its underlying classifier is. Nevertheless, if saliency method often hits the bounding box it is reasonable to conclude that both the classifier and the saliency method behave in an intuitive way, from human perspective. Our measurements in Table confirm that the ablation-path method indeed does this well.
Our method has an advantage over other ones as it yields not only a single spatial map as the saliency but a whole path of masks. However, it does not directly give saliency map. Here is how we choose one point in the image from an ablation path. First, we choose a specific time, just as we did in the examples in SG In most cases, this means that we chose the time at which the probability has dropped by 20 %, that is the smallest time such that F(t) I 0.8. When there no such time, we pick the one for which F' takes the maximum value on the path. Now that the time is chosen, we pick the point in the mask which have the smallest value. Note that, even though this selection may be unstable (since many points are close to the minimum value, the mask is typically saturated), it usually does not matter since the whole region selected by the mask is salient.
The classifier in this experiment test data set 360 images from three synsets out of ImageNet.
EfficientNet Tan & Le (2019) pre-trained on ImageNet, the
As comparison, evaluated the pointing game with the same network and inputs also through the closely related Meaningful Perturbations method Fong Vedaldi (2017). We used a hhird-party PyTorch implementation Gildenblat (2017). Note that this implementation uses slightly simplified method of removing information for performance, and that we did not change any of the hyperpa- rameters the method, is very likely that these results are not optimal. We also note that Fong & Vedaldi (2017) themselves include results for the pointing game on different data, with better scores.
Nevertheless, the point was to show that our method works reasonably well, and this may be due the use of ablation paths: that increasing family of mask probably gives some stability to the saliency method, which improves the scores.
E Random ablation paths
In this section, following an idea of Sturmfelss all (2020), compare our method random ablation-paths. Figure 2 includes one such example the random path, which has a quite fuzzy class transition in roughly the middle of the path. Figure 10 shows that this typical behaviour, In
unsmoothed
40
20
smoothed
40
20
0
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Figure 10:
Histograms of the scores of ablating an image obtained as explained in $ 4.2 (the gold finch from Figure 3 against its blurred baseline) along random paths.
14

---

Under review as a conference paper at ICLR 2021
the unsmoothed case, the scores almost compactly clustered in the middle, i.c., paths consisting of random single-pixel transitions rarely have exceptional scores. the random paths are spatially smoothed, outliers become more likely (the paths could by coincidence mask out whole particularly relevant region), but the scores are still really unlikely to reach 0.9, something our optimisation approach routinely attains.
More examples
We provide in Figure 11 series of ablation path results for class in the ILSVRC14 datasetRussakovsky et al. (2015).
15

---

Under review as a conference paper at ICLR 2021
oscrsedd Fall Fiaar p-d)
cndd Pats Traas
movn Path Fiber gat)
eeteet Park Viler pa1)
rount Patk 2ib caly
Pekeact Pats Vikee
SADRFNN
SANOFON
Tud Rano (im cal)
Deksout Path
Ostmidd
Octimied
optimbed
Optibidd e-d1
Optimbed
Octibee
02
2011
2014
Figure 11: Some extra examples with EfficcientNet and one class in the ILSVRC14 dataset. some cases, the network does not correctly classify the image, which explains the poor score.
16


Reviews:
[{"review_id": "MYefZVtrAwU", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nSaliency problem for black-box classification is the main focus in this paper, which means to find out  the part of the image that is most relevant for the current model decision. Authors propose to find an optimal ablation path between two images to get such saliency maps. The finding in this paper suggest a new view based on ablation path optimization. Several examples are presented to show the behavior of the proposed method for one image classification model.  \n\nPro:\n+ Treating the saliency problem as finding an optimal ablation path between two images is interesting. Formulating the saliency problem as an optimization problem on paths is a kind of new. \n+ Reasonable results can be obtained on several example images to show the decision of the model.\n\nConcerns:\n- The proposed ablation path saliency method requires the existence of example pairs, which limits the application of the method. It seems cannot handle the case if we are only interested in the saliency map for one single image without any support images (baseline image). \n- It is also not clear how to select baseline images from a large image dataset (e.g. ImageNet) to get the optimal saliency map for one target image.\n- There is a lack of quantitative evaluations to validate the effectiveness of the proposed method. Only showing several examples based on fixed image pairs is not that convincing. Why not follow some existing evaluation tools to conduct quantitative evaluations and compare with existing saliency methods? Evaluations could be Pointing Game or weakly-supervised localization/segmentation as in Grad-CAM (Selvaraju et al. 2016) as well as the sensitivity-n property proposed in (Ancona et al. 2017).\n- Only examples for image classification models are presented. It is not clear whether the proposed method can be applied to other problems such as neural machine translation, question classification (Sundararajan et al. 2017), image captioning (Selvaraju et al. 2016) and etc. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ln02wIJR2HD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors cast the saliency problem as finding an optimal ablation path between two images.\n(1) According to the formulation of the saliency problem proposed in this study, the authors try to connect previous notions of saliency, notably integrated gradients and ablation tests via guiding where to add regularisation.\n(2) One of my main concerns is that although the effect of the proposed strategy is illustrated based on several instances, more objective evaluation of specific tasks on large scale datasets needs to be compared to show if the formulated idea really works. \n(3) As is known, the saliency mechanism is closely related to concepts in the field of neuroscience, so it would be better if the proposed formulation could be discussed based on its essences of cognitive neuroscience mechanisms.", "extra_scores": {}, "extra_text": {}}, {"review_id": "BiCPDWtmVR7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes the black box approach to saliency prediction using ablation paths gradually replacing the parts of the image of one class with the image of another one. The idea has certain novelty, but the reviewer cannot see that it is backed by the evidence. The following comments are describing the reasons for the current paper rating and must be addressed for the rating to be improved:\n1. The main concern of the reviewer about the paper is the lack of experimental evidence: it only provides the results for use cases and no quantitative assessment over the larger dataset.  For example, Fong&Vedaldi [1] assessed the method of 50K ImageNet validation images. As far as the reviewer can see, there is no qualitative assessment on a sufficient number of samples which would prove the efficiency and compare the method against the alternatives. This concern is critical for the given score of the paper.  \n2. Besides of that, it would be also important to emphasise the novelty: why do we need to have another saliency method, does it give any new insights other than the existing ones mentioned in the related work section? \n3. It is also said in the paper: \"We performed a small, non-rigorous statistical study suggesting that scores > 0.9 are p < 0.01 significant against a null hypothesis of smoothly random masks, and extremely unlikely with pixelwise-random masks.” It would be beneficial to see some further details of it as it could improve the analysis\n4. In a light of the following description and the  given details of the optimisation method : “Since the optimisation problem is constrained (since φ is constrained by the requirements in Definition 3.1), following the gradient direction will lead us to violate the constraints. Since the constraints are convex, it is straightforward enough to project each gradient-updated version back to something that does fulfill them, and indeed that is the idea behind our algorithm, however in practice it does by itself not yield convergence without impractically many iterations” it is not clear, what the theoretical justification of it is. How do we know that the proposed artificial saturation could be generalised to other samples of data, beyond the use cases? \n5. Further to that, the paper claims black box optimisation approach but the tests have only been reported for  Inception v4 pretrained on ImageNet. Would It still work if we use something different from it? \n6. In a few qualitative examples given, the method is compared only to Integrated gradients. Is it possible to add a comparison with other methods,  e.g. Fong&Vedaldi[1] which is mentioned throughout the text as a closely related method?\n \n[1] Fong&Vedaldi (2018) Interpretable Explanations of Black Boxes by Meaningful Perturbation", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the reviewers in general liked the ideas proposed in the paper, the experimental evaluation has several issues that need fixing before it can be accepted.
Decision: Reject