Distribution-Guided Local Explanation for Black-Box Classifiers
Authors: Weijie Fu
Meng Wang
Mengnan Du
Ninghao Liu
Shijie Hao
Xia Hu
Venue:  (2020)
Keywords: explanation
cnn
saliency map

Abstract: Existing local explanation methods provide an explanation for each decision of black-box classifiers, in the form of relevance scores of features according to their contributions. To obtain satisfying explainability, many methods introduce ad hoc constraints into the classification loss to regularize these relevance scores. However, the large information gap between the classification loss and these constraints increases the difficulty of tuning hyper-parameters. To bridge this gap, in this paper we present a simple but effective mask predictor. Specifically, we model the above constraints with a distribution controller, and integrate it with a neural network to directly guide the distribution of relevance scores. The benefit of this strategy is to facilitate the setting of involved hyper-parameters, and enable discriminative scores over supporting features. The experimental results demonstrate that our method outperforms others in terms of faithfulness and explainability. Meanwhile, it also provides effective saliency maps for explaining each decision. 

Under review as a conference paper at ICLR 2020
DISTRIBUTION-GUDEE LOCAL EXPLANATION FOR BLACK-BOX CLASSIFIERS
Anonymous authors Paper under double-blind review
ABSTRACT
Existing local explanation methods provide explanation for each decision of black-box classifiers, in the form of relevance scores of features according to their contributions. To obtain satisfying explainability, many methods introduce ad hoc constraints into the classification loss to regularize these relevance scores. In this paper, we discuss some shortcomings of these methods and address them by a simple but effective mask predictor. Specifically, we present the concept of dis- tribution controllers and integrate it with neural network directly guide the distribution of relevance scores. Then we introduce the classification loss to opti- mize the proposed predictor. The benefit of this strategy to enable discrimina- tive scores over supporting features, and facilitate the setting of involved hyper- parameters. The experimental results demonstrate that the proposed method also outperforms others in terms of faithfulness and explainability. The code is avail- able at httpss///gthhubbcc///cccllllllll
INTRODUCTION
Deep neural networks (DNNs) have achieved high classification accuracy in wide range of fields, such as computer vision (He al. 2016, Simonyan & Zissermann 2014) and natural language pro- cessing (Greff et al., 2016 Mikolov et al.) 2010). Despite the superior performance, DNN models lack meaningful explanations on how a specific decision is made, and are often regarded as black- box classifiers. To address this issue, various global and local explanation methods have been pro- posed. The former group aims to inspect the structures and the parameters of a complex model (Erhan et al.] 2009 Chen et al., 2016). The latter group provides users understandable rationale for specific decision with relevance scores (Simonyan et al. 2014 Du et al. Sa).
In this paper, we focus local explanation as it extracts the intuitive evidence behind the decision of each instance. To obtain the relevance scores for local explanation, gradient-based methods com- pute the partial derivative of the class probability with respect to an input instance. However, instead of directly pointing out why the target class is derived based on input, it is likely to answer the question (Montavon al. 2018): What makes this instance more or less similar to the target class? To tackle this limitation, perturbation- based explanation methods are proposed. These methods per- turb the input and aim to find the smallest region, which alone allows confident classification or prevents a confident classification once being removed (Dabkowski & Gal 2017, Fong & Vedaldi 2017). By applying various ad hoc constraints, these methods improve explainability and maintains faithfulness ? Nevertheless, most of these methods either focus distinguishing the supporting features from background and ignore the discrimination on these features, or are only able to pre- dict discriminative scores small part of supporting features. To understand the behaviour of classifier, the discriminative scores over all supporting features are preferred. The comparison of an expected salinecy map others are shown in Fig. [ Moreover, the information gap between the classification loss and the constraints in turn increases the difficulty of tuning hyper-parameters.
Inn this paper, relevance scores indicate the contributions of features to a specific decision. A high score implies higher contribution. Besides, we not discriminate saliency maps and masks, as both indicate the permutation of relevance scores of each input.
Explainability quantifies how easy it is understand and mates the fidelity between the explanation and the decision hehaviour of black-box classifiers.
ahout the explanation; faithfulness esti-
Bessddes highlighting the pixels within the dog for the images of dogs, the relevance scores should also be discriminative various parts of the dog, such higher scores the head than legs.

---

Under review as a conference paper at ICLR 2020
Focodar Black-box Classifier
layer
Layer 3
Layer 4
Layer S
(b)
right rail
Upsample
Loss for Closs
CQM))
adf(nn
pdf(c)
Feature Filer
Dimiisvtionn Tranefo mer
Nornaliaation
(c)
50"(0)
Monie from right-lreweed lafs ckeved cistribultin controllers.
Distributtion Controller C
Mask Generctoor
C
Figure (a) An example the comparison of the expected saliency map (the last one) to others. (b) An example of right-skewed distributions. (c) The illustration of the benefits of using right-skewed distributions for explanation. (d) The framework of our mask predictor, where a distribution controller is introduced right after the mask generator. In particular, pdf stands for the probability density function.
To address these issue, we propose a simple but effective mask predictor. The work is built upon the following observation: large portion of contributions each decision are held by only small fraction of features (Fong & Vedaldi 2017) Chattopadhay et al. 22118] Du et al. 2018a). The proposed predictor consists of mask generator and distribution controller. The former takes the hidden feature maps in black-box classifiers as inputs, and the latter guides the outputs with expected distributions relevance scores. We show that, with an easy setting of the involved hyper- parameters in the controller, it can directly enforce the relevance scores towards right-skewed distri- butions (Clauset et al., 2009). Then, scores of supporting features becomes more discriminative, which correspond to the small portion of the right tail, and the majority features have low scores and are regarded as unimportant. For illustration, an example in (bb), and the masks predicted based on the distributions controllers with the right-skewed to the left-skewed are displayed in Ic). Furthermore, we introduce classification losses to opti- mize mask predictors, which removes all constraints and avoid non-trivial hyper-parameter tuning. We show that such simplifications will not sacrifice the explainability too much with an ablation study. Finally, we introduce two metrics for comprehensively evaluating relevance scores in terms of faithfulness and explainability, respectively. The main contributions of our work are as follows.
right-skewed distribution is shown
We present the conception of distribution controllers on relevance scores, and integrate with trainable mask generator improve the discrimination of supporting features. We provide two practical implementations of controllers to enforce scores towards the desired right-skewed distributions, where the involved
can be casily set.
We introduce the classification loss to train the proposed model. avoids the non-trivial hyper-paramedt tuning on ad hoc constraints and also improves the faithfulness of mim- icking target black-box classifiers.
We empirically demonstrate the effectiveness of the above innovations. Specifically, change the setting of distributions and show the controllers can guide the scores towards varying preset distributions. Besides, perform ablation study by adding constraints back to analyze their effect masks. Furthermore, the experiments also demonstrate that our method only obtains higher quantitative performance for explaining the behavior of black-box classifiers, but also provides discriminative masks for intuitive explanation.
RELATED WORK
This section reviews local explanation methods for DNNs. which target to identify the relevance score of each feature towards specific decision Moontaon al. 2018).
2.1 GRADIENT-BASEE BASED LOCAL EXPLANATION
To obtain relevance scores, gradient- based methods compute the partial derivative the class prob- ability with respect to the input by using back pocpppropagttion (Simonyan al. 2014). In general, these
2

---

Under review as a conference paper at ICLR 2020
methods are advantageous in their high computational efficiency, ie., using few forward and back- ward iterations is sufficient to generate saliency maps. However, these saliency maps based on the naive gradients visually noisy and hard to be understand. To address this issue, various methods have been proposed. For example, Smooth Grad (Smilkov et al.] 2017) addresses the visual noise by introducing noise to inputs repeatedly. Integrated gradient (Fong & Vedaldi (007)) estimates the global contribution of each feature rather than the local sensitivity. Guided back- pooc--rrogattion (Sprin-l genberg et al. 2014) modifies the gradients of RELU functions by discarding negative values the sock-propagation process. Besides, recent methods proposed to create saliency maps by combining the gradients with the corresponding features. For example, Grad CAM (Selvaraju al.) (001)) and Grad CAM ++ (Chattopadhay et al., 2018)) take advantages of high-level feature maps, which makes their saliency maps more clear. Nevertheless, since black-box classifiers are trained without any location information, the object locations in their high-level layers may not always correspond to the locations in raw images, leading to new issues for these explanation methods.
2.2 PERTURBATION-BASEE LOCAL EXPLANATION
The perturbation- based methods first perturb an input according to a given mask, and then observe the new class probability of black-box classifiers. By measuring its difference to the probability of the raw input, the supporting features in the input with the class label can be located (Dabkowski & Gal Gal,2011 2017). Let M indicate the expected mask of image where my EM indicates the relevance score of the pixel within I. Besides perturbing with mask M, these methods also introduce an alternative background image A to reduce the amount of unwanted evidences:
(1I, = I o M + A ④ (1.
Suppose fe indicates the probability of the predicted class e based on the black-box classifier. These method input the perturbed images to the classifier and optimize the mask with ad hoc constraints:
argming Ex (M) = argminm Aav AVM + Arv TV(M) fe(o(I, + A 1 M))
(2)
where AV aims to minimize the average of all scores and TV enforces the mask to smooth. The last two terms aim to obtain discriminative scores between supporting pixels and the rest pixels.
To improve the explainability, various novel methods following the above formulation are subse- quently proposed. For example, (Du et al. 2018b) regularizes the expected mask with middle-level features and optimizes the mask by reconstructing higher-level feature maps. (Fong & Vedaldi 2017) introduces deletion game and reformulates the problem by applying multiple masks stochas- tically. (Fong et al., 2019) introduces extreme perturbation with an area constraint. However, these methods need to optimize the mask for each image individually, leading non-negligible time costs.
THE PROPOSED METHOD
In this section, we introduce the details of the proposed framework, as shown in 1 It consists of encoder inside the black-box classifier F, a mask generator G, and distribution controller C. In particular, the generator and controller together compose the mask predictor, which takes the fea- ture maps of the black-box classifier as inputs and predicts the relevance scores for each instance. In the following sections, we first review the mask generator and then introduce the proposed predictor.
3.1 THE REVIEW OF MASK GENERATOR
We first describe the mask generator G in our framework. Specifically, it contains three bottleneck blocks and follows the U-Net architecture with the black-box classifier (Dabkowski Gal 2017). Each image is input into the classifier for producing feature maps multiple layers. Then the mask generator upsamples the feature maps of lower resolution using transposed convolutions, and con- catenates them with the higler-resolution feature maps. Based multiple transposed convolution layers, the generator produces a reduced mask MF=(G(F(I)) at a coarse scale and obtain the rel- evance score each location. Then, upsampling based on bilinear interpolation is employed to obtain more smooth masks at the image seale. Let M indicate the expected mask. Denote F(I) as the feature maps for the image I. The image is then perturbed with the generated mask with 1q.
3

---

Under review as a conference paper at ICLR 2020
(a)
(c)
Figure 2: The examples of the ablation studies on (a) Daakkoski Gal 2017) and (b) the proposed method.
(c) The negative effect of non-monotonic transformers.
and the generator can be optimized based on the constraints in Eq.2:
argming6R> (M), where M = upsample(MF).
(3)
In particular, simple feature filter is pretrained to perform initial localization with respect to the predicted class. Since it is not the focus of this paper, we leave the detailed description in A.
The generator module produce explanation masks in real time. However, the problem of this method is that, it only tries to distinguish the supporting features from others, without considering the discrimination of scores among these supporting features for better explainability. Besides, bal- ancing the trade-off's between the classification loss and additional constraints, such as the smoothing term Eq. 2) involves a non-trivial hyper-paramenn tuning process.
For illustration, an example is shown in F(g.2(a), where the masks are obtained from the trained gen- erators under the following hyper-parann arper-paraetter settings: (1) prrefully-tuned Äs, (2) A (3) Abg=0, (4) =0, (5) =0.== As we observed, each term in Eq. 2 brings a significant effect on final masks. Of note, although Ang=0 can release the masks to obtain discriminative scores on small part of supporting pixels, it reduces the explainability on the others. Besides, it increases the sensi- tiveness of Agv on masks. The reason is that by enforcing fe( to low values, masks can highlight the majority of supporting pixels even when λav is slightly larger. Once it is removed with A0, the size of high scores is totally controlled by Aay More details are discussed in Sec. 4.4.
3.2 THE MASK PREDICTOR
To address the above issues, we first present distribution controller after the above mask generator to guide the relevance scores. Then, we introduce classification losses for optimizing the predictor without hoc constraints. The details two technical innovations are presented below.
3.2.1 THE CONTROLLER ON DISTRIBUTION
To produce discriminative scores on supporting features in M, we introduce the conception dis- tribution controllers C, which is used to guide the relevance scores towards desired distributions. Suppose M the output of the generator on I, and Md denotes the output of C. The output of distribution controller is expressed as
M = C(M+).
Based on the above transform, we expect to encode the distribution inside the mask predictor rather than bringing the formulation an additional constraint. Besides, we follow (Dabkowski & Gal) 2017 Du et al. 2018b) to upsample M with interpolation, aiming to improve the smoothness of masks.
The principles of controllers. Now we investigate two principles for designing the controller C.
Principle We expect that the output scores of approach a right-skewed distribution for each instance. The motivation behind is that a large portion of contributions to each decision are sup- posed to be held only by a small fraction of supporting features with varying importance. These supporting features, whose information should be preserved by the mask, assigned with higher but discriminative scores.
We impose the monotonic mapping from the distribution controller's input M to its output Md. The purpose is to enhance the discriminative ability of the coarse mask M but without changing its ranking of relevance scores.
Principle 2.
For better illustration, two examples are shown in Fig. (a) and Fig.2(c). In the first example, with the expected distributions of controllers varying from the right-skewed to the left-skewed, the dis-
4

---

Under review as a conference paper at ICLR 2020
crimination of supporting features is remarkably reduced. Besides, the proportion of features that are highlighted with high scores is nearly positive correlation to the area at the right part of preset distributions. In the second example, non non-monotonic transform g(æ) x² is used in controllers, which results the scores in the deviance from expected meanings. That is, a higher score implies a larger contribution.
The implementation of controllers. Following the above principles, we introduce two piratical implementation of the controllers. Since sum of even two independent random variables is more Gaussian than the original variables (Hyvärinen & Oja 2000 loffe & Szegedy) 2013), we first assume that the imput scores of controllers (the outputs of generators) are nearly Gaussian for conve- nience. Later we show that the proposed controller built upon this assumption can also benefit other distributions towards the right-skewed.
From the Gaussian to the normal. In order to obtain the desired distributions, we first introduce instance normalization (Ulyanov et al.] (016)) to transform Gaussian distributions towards normal distributions. The goal of this step is to shift scores around two sides of O.
Specifically, let m EMs indicates the input entry at the location (i..)) of C, and Ez denotes the expected variable with a normal distribution. The mapping can be expressed as
Zij = (m² EEmfj)//
Var[m)))
(5)
where the expectation E[-] and variance Var[-] are computed over the entries of each M
From the normal to the right-skewed. Now further transform the above scores towards right- skewed distribution monotonically. To do this, we introduce two simple transformers, including a basic one without any hyper- param-er,,eeer, and customized one with easy-to-set hyper-parameters.
The straightforward way of transforming a normal distribution into a right- skewed distributions is cropping variables with Rectified Linear Unit (ReLU) (Nair & Hinton 2010). Specifically, the expected output of a distribution controller mi, can be obtained as
Basic Transformer (BT):
m ReI.U(z;)) = max(0, z:y))
(6)
The benefit of BT is to enable a monotonic transformation without any hyper-parameter. However, BT is lack of the potential ability to handle other distributions, such as the left-skewed inputs.
Customized Transformer (CT): To address the issues of BT, we introduce an alternative with easy- to-set hyper-parameters. The goal is to produce scores for all features with a right tail in (0,1). Specifically, we first transform the normal distribution towards an uniform distribution based on sigmoid functions, and then change the skewness of the distribution based on power functions:
(sigmoid(n) z1z))* = (1
(7)
+1+exy exp 7:2
where 17 is used approach the uniform distribution (Wasssi & Rossin 1996), and h determines the skewness of the obtain distribution. Based on the probability density transformation (Forbes et al.) 2011), we can easily set these hyper-parameters based on the intuitive geometric properties of their probability density functions. For example, with 1p=1.5 and h=2.5, we can obtain the scores with the probability density function in 1 Owing to the space limitation, we leave the detailed discussion to A.2 Note that other transformers with monotonicity can also be considered.
The affect on other distributions. Next, we release the Gaussian assumption and consider the effect of the preset controller on other distributions. Specifically, we introduce the CT with 1751.5 and h=2.5 on these distributions. The original (in gray) and transformed distributions (in blue) are displayed in Figs.3(a-d) As we can see, although CT may not transform them to the right-skewed perfectly, it can still shift a part of scores to low values and make scores away from the skewes.
The post hoe tuning of distributions. In practice, constraining different images with same dis- tribution may lead to redundant scores on unimportant pixels, especially when the supporting pixels only take small part of the image. For this, we present simple trick perform post hoc tuning.
Given a predicted mask, which ranks all image pixels with respect to their scores for the predicted class. We first iteratively add ranked pixels from each image and measure the resulting probability for pixels. Next, we modify these probabilities with q==axx lizi to ensure the monotonic- ity. Since the pixel i is not likely be the supporting one if qi already approaches the maximum
5

---

Under review as a conference paper at ICLR 2020
(P)
(
Figure 3: (a-d). The effect of CT on different distributions. (e) The effect of post hoc tuning on masks.
maxx= 4s we post hoc tune the relevance score as r/,r,xu,, where oy= 2h +eps, and r. original predicted score of the i-th pixel. An example is shown in Fig.3(c), where post hoc tuning performed on the masks that obtained from BT (the top two) and CT (the bottom two).
note, benefiting from the enforced monotonicity, the ranking of relevance scores can remained. Besides, we uniformly sample a limited number pixels for estimating the probabilities and infer the remaining with linear interpolation.
3.2.2 OPTIMIZATION WITH CLASSIFICATION LOSS
In this section, we introduce classification loss to optimize the above mask predictor. Denote fci the classification loss used for training the black-box classifier the proposed predictor can be optimized without the non non-trivial hyper-parameter tuning as
(fl c), where M = upsample(C(Ms)).
(8)
Since the predictor formulated to find the region that maximizes the target classification under the expected distribution, it can also better mimics the black-box classifier.
observed that when the cross entropy loss is used for training the original classifier /. Eq g similar to removing all ad hoc constraints in Eq
Thus, one may argue that the abandon these constraints may remarkably scarify the quality of masks. For this, we provide example of the ablation study by adding all constraints back and then display the masks from the trained predictors Fig.2(b) under the following hyper-parameter
settings: (1) carefully-tune λs, (2) A0==0, (3) = (4) =λ,== (5) >av= >ay=Abg=A\, In general, we still obtaín satisfying masks without any constraint. Detailed analysis is left to Sec. 4.4.
EXPERIMENTS
This section investigates the performance of the proposed method. We firstly evaluate the faithful- ness and explainability with quantitative metrics. Then perform the visualization of masks for further comparison. Finally, we introduce the case study to empirically demonstrate the effective- ness of two proposed innovations, respectively.
4.1 SETUP
In experiments, to demonstrate the broad applicability, we applied our method to CNNs with three types of structures. Besides, we compare Ours(BT) and Ours(CT) with nine recently proposed explanation methods. The experiments are performed on real- real-world dataset, ImageNet. A brief introduction is given below, and details can found in A.3
Three types of CNNs. We use ResNet50 (He al. 2016), VGG19 (Simonyan & Zisserman 2014 and GoogleNet (Szegedy et al. 2015) as black-box classifiers to show the broad applicability. We load the parameters from traditional CNNs that were pre-trained using 1.2M ImageNet images.
Nine compared methods. The following methods are used for comparison: (1) Mask Generator (MGnet) (Dabkowski & Gal 2017). (2) Meaningful Perturbation (MPert) (Fong & Vedaldi 2017),
(3) Grad CAM (GCAM) (Selvaraju et al. 2017] (4) Grad CAM++ (GCAM++) (Chattopadhay
et al.
2018), (5) Feature Inversion (FInv) (Du al. 2018b), (6) Extreme Perturbation (XPert) (Fong
ct al. 2019), (7) Vanilla Gradient (VGrad) (Simonyan et al. 2014), (8) Smoothness Gradient (SM-
Grad)
(mmikoov et al. 2017), (9) Integrated Gradient (ITGradd (Sundararajan et al. 2017). In par-

---

Under review as a conference paper at ICLR 2020
ticular, all perturbation- based methods apply the same strategy to ours for adding noise. For the hyper-parameters in these methods, we follow the setting in their papers for a fair comparison.
Implementation details. We build the mask generator with three bottleneck blocks, which takes 7x7 feature maps as the low-level input, and predicts the mask at the 56x56 in size. We use a two- stage scheme to train the mask predictor. We first train the feature filter based on 250,000 images sampled from the training set, and then optimize other parts of the mask predictor with the batch size of 64. Of note, no ground truth is introduced and only the outputs of the classifier are utilized. We use Adam for 10 epochs with the initialized learning rate of 10-? We apply step decay, and reduce the learning rate by half every three epochs. In addition, during training stage, 50% of cases the image A is the Gaussian blurred version of I with a variance of 10. The remainder of cases, A set to a random colour image with the addition of Gaussian noise.
Quantitative metries. Evaluating the quality of masks based on heuristical segmentation will reduce the fairness of comparisons. It understandable that the segmentation will be significantly affected by thresholds (Fong & Vedaldi (017)). To address this issue, we introduce two generalized metries to evaluate the comprehensive quality in terms of both the faithfulness and the explainability.
Faithfulness. The explanation is expected to accurately replicate the model's behaviour. Here we rely on the smallest sufficient region of the image that alone allows a confident and consistent clas- sification. Since it is subjective to decide how much confidence is preferred for specific decision, we extend the evaluation into a general case by taking advantage of ranked pixels.
Specifically, we first estimate the class probability of clean image as qo Then we iteratively add i ranked pixels with decreasing scores and measure the resulting probability qi of the target class, until the cully-perturbed images qp. Then, preserving curve can be generated. The increased area under the curve with the intervals of A is used as a measure of faithfulness, calculated as
Mr Σ
%
A) x 100%.
go OF
An extra discussion between My and perturbation-
methods are provided in A.4
Explainability. For quantitative evaluation of explainability, generally use the extra information such as bounding boxcs. For example, in the pointing game, the maximum point is first extracted from each generated mask. Then, according whether the maximum point falls in one of the ground truth bounding boxes or not, a hit NHis or a miss N Missse is counted. The localization accuracy of the pointing game for each object category is defined as:
NHits
Mpg = NHit + NMisses
X 100%.
(10)
This process is repeated for all categories and the results are averaged as the final accuracy. However, Mpg only unitizes one pixel for evaluation, leading supervised object localization considers a boarder pixels by calculating the intersection over union between binary masks and images. Nevertheless, still faces the issue of the choice of thresholds.
large bias. To address this issue, weakly-
Thus, we introduce a new metric by regarding relevance scores as the results of retrieval tasks. Specifically, we denote the precision as the fraction of the pixels retrieved within bounding boxes. Recall denotes the fraction of the within-bounding-e computing a precision P. and recall R; at each position in the ranked scores of pixels, we can evaluate the explainability of scores by the area under the precision-recall curve:
pixels that are successfully retrieved. By
MI = Σ P. (Ra RA-1) x 100%.
(11)
4.2 QUANTITATIVE EVALUATION
On the faithfulness with My. To evaluate the faithfulness of the proposed predictor, we calculate the mean metric MM based on 10,000 sampled images from the validation set. We set 2= as the interval. Besides, we simply add smoothed mask over the original one with a tiny weight, introducing the ranking for pixels with zero scores. We perturb images with their Gaussian blurred version based on their masks and estimate the average faithfulness. In order to avoid outliers, introduce min-max normalization on Mys over all methods for each image. Of note, although post hoc tuning remains the ranking of pixels, it is not performed for avoiding the ambiguity.
7

---

Under review as a conference paper at ICLR 2020
My
Mr
Myp
ResNet
VGG
Google
MEAAN
ResiNet
VGG
Goooge
MEAN
Reihet
VGG
Glogle
MEAN
DenTRT)
77.32
77.70
88.63
OassCTT)
81.42
78.05
N7.23
MGrel
70.68
68.52
$6.54
65.25
82.62
83.51
81.87
82.67
89.13
89.37
89 40
89.30
MPer
47.52
49.04
41.56
46.04
76.82
74.80
72.26
74.63
82.39
80.34
78.55
80.43
XPert
65.13
64.06
59.37
78.73
80.93
71.26
76.97
77.72
79.38
77.31
78.14
OCAM
75.43
67.08
67.66
70.06
79.28
74.91
82.15
78.79
78.33
84.67
88.
83.98
GCAM++
74.18
73.63
64.69
70.63
82.86
84.46
83.20
83,51
84.11
90.30
89
87.5 88
VGaa
38.03
19.81
73.17
88.67
mord
31.47
50.80
52,02
44.76
66.92 92
66.74
63.25
63.64
83.47
89.39
87 87.32
86.73
simpliiity ResNet, Google
ResNett0o, VGG19
respecively.
1001
Mf
65.00
M,
Mp
00.00
60.03
15.00
4000
2002
Reehar50
VGG19
MEAN
Radiu50
VGG18
Googlehet
MEAN
Restlet0
VGC19
Googlehiai
MEAN
(urs(BT)
Ourg(CT)
= MGne
MPrrt
Finv
XPart
AGCAM
GCAM
VGrad
SMBrad
ITGraa
Figure 4: The quantitative results of eleven methods on three CNNs. (left) Performance on M,, (middle) Performance Mzz (right) Performance on MPg. Detailed values are listed in the above table.
The results of average My are displayed in Fig.4(left) Based the results, the following observa- tions can obtained. Firstly, VGrad, SMGrad, and ITGrad are generally worse than others with a large gap. It is understandable that, these three methods only search sensitive pixels with gradients, the pixels with high scores will discretely appear in each image. As a result, these methods become harder gather the sufficient supporting information in a local receptive field and reach high class probability. Secondly, GCAM obtains comparable performance to GCAM++. Besides, Flnv that regularizes masks by utilizing intermediate layers outperforms MPert. suggests that, without us- ing the lowest resolution feature maps, an effective regularization is of importance for individually optimizing the mask of each instance. Thirdly, our methods outperform most compared methods, and obtain much higher performance than MGnet. Consider that the three methods apply the same architecture of mask generators. Thus, the results demonstrate that guiding the relevance scores towards right-skewed distributions enjoys better ranking of supporting features.
On the explainability with My- To reveal the explainability, we first introduce MM to evaluate the performance of object location. For this, resize and crop bounding boxes to the size of 224 x24, leading to the same size of test images. The experiments are performed on 10,000 validation images with bounding box annotations.
The results average My are displayed in Fig.4(middle). From these results, we have the following observations. Firstly, GCAM++ consistently obtained much better performance than GCAM. understandable that the former is able to detect multiple objects the image and assign them high relevance scores. Secondly, by replacing all constraints with a simple distribution controller, Ours(CT) outperforms MGnet with a small gap, and both of them enjoy better performance than most of other methods. The reason is that, benefiting from the training with large-scale images, their predicted high relevance scores focus on objects more robustly. Thirdly, the last three gradient- based methods perform worse than others. The possible reason is that, gradients are insensitive to the smooth supporting regions, which makes these regions ignored and reduces the performances.
On the explainability with Mpg. In this experiment, we show that our methods still bring com- parable performances in pointing games. The average results of all compared methods on pointing games are shown in Fig.4(right). From the results this figure, the following observations can be obtained. Firstly, SMGrad and ITGrad now obtain comparable even better performance than the first eight methods and outperform VGrad. The main reason is that, this metric only considers the localization precision of the top pixel rather than group of pixels. By removing noise in gradients to some degree, these two methods are able to locate the most important pixel. Secondly, although methods remove all constraints, e.g. 00= and =0, they still obtain the comparable accuracy
8

---

Under review as a conference paper at ICLR 2020
Fmages Cursdir) DusjjCt) kilret
MPvt
few
XPrrt
SCAM
OCAM
Vinia
Images
Sefere/Ater
Figure (left). Masks of different methods with ResNet50. (right). The effect post hac tuning.
of locating the most important pixel. One possible reason is that, by optimizing the predictor with training data, a large number of noisy features can removed, Ieading to the robust estimation of scores. Thirdly, MGnet enjoys the best performance. It means that, by enforcing the background towards a low prohability, the high scores on the background are generally prevented.
4.3 EXPLAINABILITY WITH VISUALIZATION
Comparison to baseline methods. We qualitatively compare the saliency maps produced using our methods with those produced by other baselines. The red and blue denotes the high and low score, respectively. In this experiments, no post hoc tuning is used for the proposed methods. We randomly sample images from ImageNet and show their masks obtained with ResNet50 in Fig.5(left). More examples from other CNNs are displayed in Figs 8 12
in
A.5
From these results, we have the following observations. Firstly, the gradient- methods gen- erally results in more high scores outside the objects. Secondly, although the high-level feature maps are used to build saliency maps, GCAM and GCAM+ may still miss the supporting pixels of objects, such the last second row in Fie.5(left). Thirdly, MGnet obtains high scores the supporting features. However, it only assigns them similar relevance scores. Finally, compared with other methods, with easy setting of hyper-parameterr Ours(CT) not only predicts higher scores on these pixels than others but also builds discriminative scores over these pixels. It shows the effectiveness of guiding scores under right-skewed distributions.
The effect of post hoc tuning. In this experiment, we aim to investigate the effectiveness of post hoc tuning improve the distribution of relevance scores. Frg.5(righh) shows examples of our masks with ResNet50 before and after the tuning operation. The masks of MGnet are also displayed for comparison. More examples of this task can be found in Fig A.5
From the figure, it is observed that compared with MGnet, our predicted masks sacrifice the differ- ence of scores between the supporting features and others to some degree. However, owing the better ranking of scores over pixels (which is empirically demonstrated by the metric MMr), we can easily utilize the post hoc tuning to remove the noise and highlight the supporting pixels.
4.4 PERFORMANCE OF THE INNOVATIONS IN OUR PREDICTOR
On distribution controller: the effect distributions. To show that the distributions of output masks are benefited from the proposed controller, study the masks from the predictors optimized under varying preset distributions. Specifically, we change the Then train mask predictors with these controllers. The resulted masks with the corresponding preset distributions are shown in Fig.6(left).
of controllers with
As observed, although is hard to expect that all masks obtain scores with the same distribution to the expected one, the controllers enforce them towards these distributions. Besides, we perform the transformer the obtained masks of MGnet in post hoc manner, which fails to build discrim-

---

Under review as a conference paper at ICLR 2020
Figure 6: (left) The effect preset distributions (Ist row) on the predicted masks (2nd-8th rows). (right) The ablation study on the regularizations of different methods.
inative scores supporting features. The reason is understandable that the scores are already be stacked at the values of 0 and
On optimization with classification loss: the effect of regularization. To evaluate the impact of our simplified objective function, we introduce an ablation study to analyze how the regularization terms affect our mask predictor. We add all ad hoc constraints back to the formulation and train the
mask predictor using the following hyper-parameter settings: (1) carefully-tuned tuned As, (2) Aa==0, (3)
\av=Abg=O, (4) =0,==000 (5) > For comparison, ablation study is also perform on MGnet with the setting of (1) carefully-tuned Ås, (2) Aav=0, (3) A0,= (4) A0= (5) =0.=0. The masks from the trained models with varying settings displayed in Fig.6(right).
By adding all ad hoc constraints back, the masks of Ours(CT) can be improved to some degree. However, we observed that A===0 has few effect on our predictor. In addition, Abg=0 will increase the noise around the boundary owing to the smoothness constraint, and Asv=0 causes the holes in the masks. Nevertheless, by further removing these two terms, these shortcomings can be alleviated and the final masks of Ours(CT) still remain satisfying quality. On the contrary, besides A,= all terms result in remarkable impacts on the masks of MGnet. note, although Abg=0 can release the mask towards discriminative scores, it would reduce the discrimination features, such as the second, fourth and fifth rows. Besides, it increases the sensitiveness of An on masks. For example, while Abg is carefully set, dav varying within (2.12) consistently leads to acceptable masks for ResNet50. Once Agv=O, the number of high scores is totally controlled by Anv and the quality of masks is only acceptable with (2,44) In short, benefiting from the distribution controller, Ours(CT) is insensitive to the regularization terms and can predict satisfying masks without ad hoc constraints.
the majority of supporting
CONCLUSION AND FURTHER WORK
This paper presents simple but effective mask predictor to provide local explanations for black- box classifiers. Specifically, we introduce the conception of distribution controller on relevance scores, and integrate with a trainable mask generator directly guide the relevance scores. In addition, the proposed predictor is optimized under the classification loss, aiming to better mimic the classifiers. Based on the proposed innovations, our method enables discriminative scores over supporting features and also facilitates the setting of involved hyper- hyper-paramene parameters. The experimental results demonstrate that our method outperforms others in terms of faithfulness and explainability.
There are some aspects needing further investigations. Firstly, since the objective functions are formulated the same to those of classifiers, it is natural to train the classifiers and mask predictors at the same time. As result, can provide users pair of classifier and explainer. Secondly, although this paper provides an intuitive comparison of the transformed distributions for setting hyper-parameters, quantitative analysis the proportion of features the tail could be studied.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- cam++ Generalized gradient-based visual explanations for deep convolutional networks. /EEE Winter Conference on Applications of Computer Vision (WACV), pp. 839-847, IEEE, 2018
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), pp. 2172-21800 2016.
Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. Power-law distributions in empirical data. SIAM review, 51 (14)6661) 2009.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Advances in Neural Information Processing Systems (NeurIPS), pp. 6967-6976, 2017.
Mengnan Du, Ninghao Liu, and Xia Hu. Techniques for interpretable machine learning. arXiv preprint arXiv:/808. /00.000338,,,333,, 2018a.
Mengnan Du, Ninghao Liu, Qingquan Song, and Xia Hu. Towards explanation of dnn-based predic- tion with guided feature inversion. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), 2018b.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.
Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal per- turbations and smooth masks. In Proceedings of the IEEE International Conference Computer Vision (ICCV). pp. 2950-29000099 2958, 2019.
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful pertur- bation. In Proceedings of the IEFE International Conference on Computer Vision (ICCV), pp- 1429-3337, 2017.
Catherine Forbes, Merran Evans, Nicholas Hastings, and Brian Peacock. Statistical distributions. John Wiley & Sons, 2011.
Klaus Greff, Rupesh Srivastava, Jan Koutník, Bas Steunebrink, and Jürgen Schmidhuber. Lstm: A search space odyssey. IEEE transactions on neural networks and learning systems (TNNLS), 28(10) 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural networks, 4(4-5)):11-4340 2000.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by re- ducing internal covariate shift. In Proceedings of the IEEE International Conference on Machine Learning (ICML), pp. 448 456, 2015.
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In The annual conference of the international speech communication association, 2010.
Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Müller. Methods for interpreting and un- derstanding deep neural networks. Digital Signal Processing, 73:1-15, 2018.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the International Conference Machine Learning (ICMI), pp. 807-814, 2010.
11

---

Under review as a conference paper at ICLR 2020
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference Computer Vision (ICCV), pp- 618-626, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. International Conference on Learning Representations (ICLR) Workshop, 2014.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenherg. Smoothgrad: removing noise by adding noise. ICML workshop, 2017.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. International Conference on Learning Representations (ICLR) Workshop, 2014.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan, Axiomatic attribution for deep networks. In Pro- ceedings of the International Conference on Machine Learning (ICML), pp. org, 2017.
JMLR.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference Computer Vision and Puttern Recognition (CVPR), pp- 1-9, 2015.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv:1607.08022 2016.
Gary Waissi and Donald F Rossin. A sigmoid approximation of the standard normal integral. Ap- plied Mathematics and Computation, 77(1):91-95, 1996.
A APPENDIX
A.1 THE FEATURE FILTER IN THE MASK GENERATOR
The purpose of the feature filter is to attenuate spatial locations whose contents do not correspond to the selected class. Denote XER XX as the output of the last convolution layer of the classifier, where H.W C' indicate the height, the width, and the number of channels of its feature maps, respectively. The output of filter YER TeR#xW> XXXC at the spatial location i, is formulated as
Y1p = X Xxjssigmoid(X..comm C.).
(12)
where C,ERIxC is the embedding of the class For efficient optimization, (Dabkowski & Gal 2017) introduces noise on real labels. Specifically, gradient- gradient-based optimizers are employed to max- imize mean(sigmoid X,,Co)) k=s and minimize mean (sigmoid XX], C.)), ksfs, iteratively.
A.2 THE SETTING OF THE HYPER- PAPEERPARAMETERS OF CT
Now we show the way of easily setting the hyper-parameters of CT based on the intuitive geometric properties of probability density functions.
To do this. we estimate the probability density function of m With the probability density trans- Foriate Forbes al. (20IT), the transformed probability density function can be obtained
formation
exp( (In(m( (1/h)) 1))
p(mm) 22mhn m(1 m11-m1/h)
(13)
2n1
where the superscript is removed for clarity. The detailed proof is provided at the end of A.2. Now set the pyree-parameters based on their effects on the geometry of probability density functions.
12

---

Under review as a conference paper at ICLR 2020
1 1 0.5
32
plm
p(m
2.0
2.5
2.0
0.2
0.4
0.5
1.0
1.5
m
0.8
0.0
0.5
h
(b)
20.07
17.5
17.5
15.0
12.5
10.0
5.0
2.5
2.5
0.0
0.0
0.2
m
0.4
66
0.0
1.5
2.0
m
0.4
86
0.0
0.5
2.0
h
(c)
(d)
Figure The probability density functions with the different settings hyper-parameters.
We fix /=1 and observe the effect of The corresponding probability density function is displayed Fig 7 a). By changing v) within (0.3,2.5), the density functions of p(m²) approximately change from the concave to the convex for 10.1). In particular, 171.55 approximately leads p(m") to uniform distribution. Considering 1.5~0.9. R is also consistent to the sigmoid approximation of the cumulative probabilities of the standard normal distribution (Waissi & Rossin 1996).
Then we set (055. and observe the effect h. Three probability density functions are displayed in (bbd). As for m >0.5, which will potentially waste this range of scores for highlighting, supporting features. Besides, owing to the strong convexity in the first step of the transformer, p(m²) with y=2..5 contin- ues this undesired property. On the contrary, benefiting from an approximated uniform distribution with 1=1.5, the distribution of p(m") in Fig 7d)) is modified from the trend of the left-skewed the right-skewed with light tail over (0, 1). Since a proportion of features corresponding to the area the tail will be highlighted with higher scores, we fix 1=1.5 and 2=2.5 to obtain discriminative scores on supporting features.
see, p(m") with 7=0.5 leads to extremely low probabilities
Proof. According the probability density transformation Forbes et al. (201T), the transformed probability density function can be obtained based on that of the original variable:
0g-5 3m
p(m) P=(9. (m))
(14)
where P-(-) is the probability density function of the original variable z, and 9.- (m) is the inverse function of m on Since m=g=(2) (2)= (T) mp)
" 1(( in Eq we obtain
1
92 (m)
In(m 1/b 1),
(15)
13

---

Under review as a conference paper at ICLR 2020
where (m-1/h-1)>00 Besides, obtain:
Øg. (m)
mm (-1h)))
1
0m
11 ml ml-1/hh
h
1
ml
(16)
1/h m ml-1/h) -
1
nh ml (1/h))
Recall the probability density function of a normal distribution is
2²
(2(2)
exp(
(17)
2
20
where a=1 in standard normal distributions. By substituting Eqs
15
17
into Eq
14
obtain
1
exp(- (In(ml-
1))
p(m) V m(1 ml1/4))
(18)
2T²2
which completes the proof.
A.3 A DETAILED INTRODUCTION OF THE EXPERIMENTAL SETTING
Firstly, we present a brief description on the experimental settings of compared methods.
Mask Generator (Dabkowski Gal 2017). which introduces the multiple constraints into the objective functions for training in our method and using its default hyper-parameters in its publicly available codes.
predictor. For this method, we use the same setting
Meaningful Perturbation (Fong & Vedaldi 2017), which performs meaningful image per- turbations and directly optimizes masks with designed constraints. For comparison, we use the same kind of perturbations in our method, and apply an Adam optimizer with the learning rate of 0.1 for optimization. The iterations is set to 300.
Grad CAM (Selvaraju et al.) (017), which upsamples the saliency maps based on the gradient-weighted weighted high-level feature maps. We use the last convolutional layers build its coarse saliency map and upsample it to the image scale as the final saliency map. Grad CAM++ (Chattopadhay et al. 2018), which conducts weighted combination of the positive partial derivatives to generate a visual explanation. We apply Grad CAM ++ with exponential functions for efficiency.
Feature Inversion (Du et al.) (0088)) which attempts to perform regularization on masks with guided feature inversion. Following their setting of papae--as,m,eers, first opti- mize the masks to reconstruct the lowest resolution features based on the second lowest resolution for 10 iterations, and then optimize the masks under the class-disrimmnnatie interpretation for 70 iterations.
Extreme Perturbation (Fong et al., 2019), which introduces extreme perturbation with an area constraint. We directly use the authors' pubic available codes for our experiments. For efficiency, the number of iteration set to 300.
Vanilla Gradient (Simonyan et al.) 2014), the basic method that uses the gradients of the raw images as saliency maps. To improve the visualization. we crop outliers and normalize the scores to [0, 1]. Smootthess-Gradee
(Smilkov et al. 2017). which removes noise by adding noise to im- ages. We apply 20% noise suggested and set the sample size to 50. Similarly, we crop outliers and normalize the scores.
Integrated Gradient (Sundararajan al. 2017), which combines the implementation invari- ance of gradients along with the sensitivity. We introduce black images as the baseline and set 200 as the number of steps in its Riemman approximation of the integral. We use the same way of improving visualization for saliency maps.
Then, we introduce the structures of mask generators for different CNNs. Specifically, for ResNet50 which consists of intermediate layers with 1.4.256.512,1024..0 channels different scales,
14

---

Under review as a conference paper at ICLR 2020
set three low-to-high-resolltiion blocks in mask generators with the channels of (256,128,64) namely, one quarter channels of the classifier; for VGG19 and GoogleNet that contain the layers with (64.128,256.512.5122 and (64.192..480.83210022 channels, half the channels in mask generators, namely, (256,128, 22,644) and a [416,240,96) for these blocks, respectively.
A.4 THE ANALYSIS ON THE RELATIONSHIP BETWEEN THE PROPOSED METRIC AND THE PERTURBATION- BASED METHODS
We provide new view to discuss the relationship between the proposed metric and wildly used perturbation- methods with the mask involved. Recall that in most perturbation-ban -baeed meth- ods, aim to maximize the probability of the target class of the perturbed image:
argmin (ot
(19)
where the pixels with higher scores in masks are supposed to be more important. By decomposing the above mask into multiple with one-hot coded mask M, Eq 19 is equal to
fe( a,M.
(20)
where oe; denotes the relevance with 0,201+1- Suppose 33 indicate some positive weights and denote a, = Bj. the above equation can further transformed into
N
fe( Σ (,)M)) fo(l,, Σ +..++ Bx)M,,
(21)
Recall the area under the probability vs. the number of pixels, which can be formulated as
(B Σ M2))) = (of + M + +M,) + 0=0
(22)
We observe that Eq 21 tends to be linear approximation of Eq 22 Specifically, two of them be- come equal when fe(.) and Φ(-) are linear. According to Eq 22 the larger values on the former 3 means the users pay more attention to the top features. According Eq it also implies the high scores should be much larger than others and highlight supporting features. Thus. to evaluate the effectiveness of perturbation-be based methods, the proposed metric tends to be more convincing.
A.5 SUPPLEMENTARY RESULTS ON VISUALIZATION EVALUATION
In this section, we provide the supplementary results visualization evaluation.
Specifically, in Figs 12 we display the predicted masks obtained from different methods with ResNet50, VGG19, and GoogleNet. By compared the proposed methods with others in these figures, we observe that our methods are generally able to distinguish the objects from the backgrounds, and also obtain discriminative scores at different parts of objects. It thus provides better explanations for the behaviours of black-box classifiers.
Besides, further demonstrate the effect of post hoc tuning, show more masks before and after the tuning operation in Fig 13 As we can see, benefiting from the effective ranking of supporting pixels, avoid the noise outside of the objects and further improve the masks.
15

---

Under review as a conference paper at ICLR 2020
Images
Curs(BT)
Ours(CT)
MGnet
MPeer
Finv
XPert
GCAM
GCAM-
VGraad
SMGrad
ITGrad
Figure 8: Examples of the saliency maps of ResNet50.
16

---

Under review as a conference paper at ICLR 2020
Images
Ours(BT)
Ours(CT)
MGnet
MPert
Finv
XPert
GCAM
GCAM-
VGrad
SMGrad
ITGrad
Figure 9: Examples of the saliency maps of VGG19,
17

---

Under review as a conference paper at ICLR 2020
Images
Ours(BT)
Ours(CT)
MGnet
MPert
Finv
XPert
GCAM
GCAM+
VGrad
SMGrad
ITGrad
Figure 10: Examples of the saliency maps of VGG19.
18

---

Under review as a conference paper at ICLR 2020
JUPAP
Images
Ourq(BT)
Ours(CT)
MGnet
MPort
Finv
XPert
GCAM
GCAM-
VGrad
SMGrad
ITGrad
Figure 11: Examples of the saliency maps of GoogleNet.
19

---

Under review as a conference paper at ICLR 2020
Images
Ours(BT)
Curs(CT)
MGnet
XPert
GCAM
GCAM++
VGrad
SMGrad
Figure 12: Examples of the saliency maps of GoogleNet.
20

---

Under review as a conference paper at ICLR 2020
Zound
MAS
DE
Images
BT (Before/After)
CT (Before/After)
MGnet
Images
BT (Before/After)
CT (Before/After)
MGnet
Figure 13: Saliency maps of the proposed methods before and after post hoc tuning.
21


Reviews:
[{"review_id": "SJxdo8dRYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduced a new framework in the family of local perturbation-based explanations (saliency maps) for deep neural networks. The method is similar to existing methods in the literature (real-time saliency maps, etc) and seeks to tackle the difficulty of hyper-parameter selection for the complex optimization objective of these methods through assuming distributional preferences over the scores in generated saliency maps. The idea is that basically many of the terms in those objectives could be achieved by enforcing a right-skewed distribution on the importance scores (and also using the interpolation idea).\n\nI lean towards rejecting this paper for one main reason: the contributions are not enough for this venue. Both the introduced method and the metrics are slight modifications of what already exists and the experimental results do not convince me that the introduced method tackles an important problem with the existing methods.\n\nMore specifically, the paper seeks to solve the hyper-parameter selection problem with existing approaches (e.g. MGNet). None of the experiments seek to show how the method is helping with this problem. For instance, the average performance of MGNet over a set of randomly selected hyperparameters versus the model's performance, the computational cost of training a good MGNet mask generator versus the introduced method, etc.\n\nOne major motivation behind perturbation-based methods is their black-box nature. The authors refer to their method as being applied to black-box models. The introduced method clearly utilizes the inner layer activations of the network. This is not a black-box method.\n\nThe main contribution is a marginally improved performance compared to rival methods for the introduced metrics (which are slight modifications to already existing metrics.) The introduced Explainability metric is very well-justified and tackles the issues with the two older metrics. The advantage of the introduced methods is not very clear on the provided subjective examples.\n\nA few suggestions and questions:\n\n* The exact contribution of the work should be stated more clearly and experiments should be targeted towards it.\n* The paper is sometimes very, very difficult to grasp (Sec 3.2 and 3.3.)\n* It might be a wrong intuition but it seems like the fine-tuning step seems to make the output scores curated for the M_F metric which would make the results in Table 1 not quite fair. The same modification seems to be applicable to any given importance mask and a more fair comparison would include applying the same fine-tuning step in all of the methods.\n* The method's optimization process seems to be only focused on the SSR and neglect SDR. The text does not make it clear why this does not result in any loss of performance and if that's the case why it was necessary for methods like MGNet.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgtwEXBYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a mask predictor to bridge the information gap between the classification loss and the constraints introduced to obtain satisfying explainability. The predictor consists of a distribution controller and a mask generator to refine the mask towards the desired score distribution. The predictor is optimized solely under the classification loss without additional constraints, which therefore improves the faithfulness of mimicking target black-box models. This paper introduces two metrics to evaluate the proposed mask predictor and the experiments demonstrate the proposed method outperforms others in terms of faithfulness and explainability.\nOverall, this paper has some advantages:\n(1) the method could be a significant contribution to improve the local explanation for black box. The mask predictor can transform the output of the mask generator to establish the right-skewed distributions for relevance scores with the designed controller, which can decrease the difficulty of tuning the hyper-parameters. To solve the problem of the ad hoc constraints, the distribution controller models these above constraints and then is integrated with a neural network to directly guide the distribution of relevance scores. \n(2) the paper develops two metrics in terms of faithfulness and explainability to evaluate the effectiveness of the proposed method quantitatively. And compared with other baseline methods, it also generates saliency maps to visually demonstrate the explainability. All the experimental results demonstrate that the performance of the proposed method is better than others.\n(3) the paper states the motivation and the proposed algorithm clearly. The code is given.\nHowever, for the experiments, the following should be addressed.\n(1) It would be better to evaluate the proposed method with other networks as black boxes.\n(2) More detailed analysis should be added on the experimental results.\n(3) It should compare the explainability with some of the latest baseline methods.\n(4) It would be better to conduct the experiments on more datasets.\n\n Moreover, there are some minor comments:\n(1) Please state more details about the innovation of the method.\n(2)\tFigure 1 needs to be polished, for example the font. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HJeWjl3VcB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Summary:\n\nThis paper proposed a method to produce instance-wise saliency map for image classification tasks. The proposed method develop a U-net-based generator for saliency mask, where the important modifications are (1) a skewness-inducing activation function for mask generation (i.e. controller), which is either a ReLU function or a scaled sigmoid function raised to a certain power. Authors argue that the proposed activation function leads to output saliency score to have right-skew distribution, which leads to more distinguished saliency map. (2) a smoothness-inducing mechanism where the saliency mask is generated at coarser scale then up-sampled with a bilinear operation so the generated map is smooth. Finally, the training is performed by minimizing solely on the cross-entropy loss with respect to the original model predictions, which is expected to improve faithfulness in mimicking the original black-box classifiers.  \n\nI'm leaning toward rejecting this paper in its current form. While I think this paper proposed an interesting strategy in improving the faithfulness of the explainer (i.e. training on the cross-entropy loss with respect to original classifier), the rest of the two modification either already exists in the literature (generate mapping at coarser resolution is an idea from Dabkowski & Gal (2017) / Du (2018) as pointed out by the author) or suffers potential technical issues that can benefit from further methodology improvement/empirical justification (please see Major Comments). Furthermore, empirical experiments seem to suggest that the proposed method sometimes fail to produce more smooth and more discriminative maps compared to its MGnet baseline (e.g. Figure 5, top right row), calling into the question of whether training solely on a cross-entropy loss is enough to ensure the generated saliency map is of high-quality. I still like the overall idea of this paper, and I think this work can be made more rigorous and informative by including a careful ablation study about the loss/gain of different regularization terms and loss functions (e.g. the effect of TV(M) term on smoothness, the effect of f(\\phi(I, 1-M)) term on discriminativeness, and the trade-off between cross-entropy v.s. the original negative log likelihood loss as used in Dabkowski & Gal (2017)).\n\n\nMajor Comments:\n\n(1) Custom Transformer\n\nThe derivation of the new activation function assumes that the distribution of the hidden-layer output $m^g_{ij}$ is strictly Gaussian (i.e. unimodal, symmetric, and with sufficiently light tail), and author proposed to build this Gaussian distribution from raw hidden-unit output through standardization (Section 3.2.2). However, this approach may not be valid in practice. This is because there is no guarantee that the distribution of $m^g_{ij}$ is strictly unimodal and symmetric. If the output is skewed/multimodal in anyway, the normalization operation won't produce a Gaussian distribution, since a linear transformation (i.e. normalization) of a skewed distribution is still a skewed distribution. As a result, I am worried that the hyperparameter derived in Section 3.2.3 may not always lead to right-skewed distribution as expected. If author would like to ensure the outcome distribution is right-skewed, it may be good to tune these activation function parameters for each instance so the empirical distribution of $m^d_{ij}$ satisfy certain skewness criteria (e.g., maybe tune the parameter in a way such that the pearson's coefficient of skewness of the empirical distribution is sufficiently small).\n\nIn addition, I find the author's claim about the benefit of right-skewed distribution (i.e. producing more discriminative maps) not yet well supported by the empirical evidence. For example, in Figure 4 the produced maps in the first three rows are relatively fuzzy, and the difference between the irrelevant features and the important features does not seem to be very high. The situation is only improved after the instance-wise finetuning. I think there are two possible reasons, either (a) the right-skewness of the actual distribution of the saliency map score in those images were not guaranteed (as mentioned earlier, I believe the current approach cannot guarantee the pre-activation units are Gaussian distributed, therefore the actual skewness of the post-activation values may vary depending on the input). (b) the right-skewness in the outcome distribution alone is not enough to guarantee good discrimininative behavior.\nTo clarify this (and provide more evidence for author's claim that skewness <-> better discrimination), author can consider visualizing the saliency map with varying degree of skewness (you can measure the skewness of the empirical distribution of the saliency-map score using pearson's coefficient of skewness), and show that the \"discriminativeness\" and skewness of the empirical distribution are correlated.\n\n(2) Smoothness of the produced map\n\nIn the original MGnet work, the authors used two mechanisms to guarantee smoothness of the saliency map: (1) generate map at coarser level and (2) TV penalty on loss function. In this work author used only (1). As a result, the produced map in some cases (e.g. Figure 5, top right row for BF After, also Figure 7, left fifth row for BF after) appear jagged and not smooth. I'm wondering if adding TV penalty back would help somewhat. Again, a careful ablation study would be beneficial in clarifying this.\n\nMinor Comments\n\n(1) There's some minor technical issue in the description for Equation (6). Assuming $z_{ij}$ is Gaussian, it is $\\eta$ AND sigmoid() COMBINED to \"approach the uniform distribution\", it is important to mention the sigmoid function since the uniform transformation used here is based on the Probability Integral Transform,   (i.e. if a random variable $z \\sim \\Phi(.)$ follows Gaussian distribution with CDF $\\Phi(.)$, then $\\Phi(z) \\sim Unif(0, 1)$, i.e. you are transforming a Gaussian random variable using Gaussian CDF to get uniform distribution), and equation (6) is using $\\eta$ AND sigmoid() combined together to approximate this Gaussian CDF [1].\n\n(2) Equation (14) on page 13. There should be an $z^2$ term within the exponential function.\n\nReference:\n[1] Gary R.Waissi, Donald F.Rossin (1996) A sigmoid approximation of the standard normal integral.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a method to estimate the instance-wise saliency map for image classification, for the purpose of improving the faithfulness of the explainer. Based on the U-net, two modifications are proposed in this work. While reviewer #3 is overall positive about this work, both Reviewer #1 and #2 rated weak reject and raised a number of concerns. The major concerns include the modifications either already exist or suffer potential issue. Reviewer #2 considered that the contributions are not enough for ICLR, and the performance improvement is marginal. The authors provided detailed responses to the reviewers’ concerns, which help to make the paper stronger, but did not change the rating. Given the concerns raised by the reviewers, the ACs agree that this paper can not be accepted at its current state.
Decision: Reject