Title: DISTRIBUTION-GUIDED LOCAL EXPLANATION FOR BLACK-BOX CLASSIFIERS

Abstract: Existing local explanation methods provide an explanation for each decision of black-box classifiers, in the form of relevance scores of features according to their contributions. To obtain satisfying explainability, many methods introduce ad hoc constraints into the classification loss to regularize these relevance scores. In this paper, we discuss some shortcomings of these methods and address them by a simple but effective mask predictor. Specifically, we present the concept of distribution controllers and integrate it with a neural network to directly guide the distribution of relevance scores. Then we introduce the classification loss to optimize the proposed predictor. The benefit of this strategy is to enable discriminative scores over supporting features, and facilitate the setting of involved hyperparameters. The experimental results demonstrate that the proposed method also outperforms others in terms of faithfulness and explainability. The code is available at https://github.com/iclrlocal/.

1. INTRODUCTION:

Deep neural networks (DNNs) have achieved high classification accuracy in a wide range of fields, such as computer vision (He et al., 2016; Simonyan & Zisserman, 2014) and natural language processing (Greff et al., 2016; Mikolov et al., 2010) . Despite the superior performance, DNN models lack meaningful explanations on how a specific decision is made, and are often regarded as blackbox classifiers. To address this issue, various global and local explanation methods have been proposed. The former group aims to inspect the structures and the parameters of a complex model (Erhan et al., 2009; Chen et al., 2016) . The latter group provides users understandable rationale for a specific decision with relevance scores1 (Simonyan et al., 2014; Du et al., 2018a) .

In this paper, we focus on local explanation as it extracts the intuitive evidence behind the decision of each instance. To obtain the relevance scores for local explanation, gradient-based methods compute the partial derivative of the class probability with respect to an input instance. However, instead of directly pointing out why the target class is derived based on input, it is likely to answer the question (Montavon et al., 2018) : What makes this instance more or less similar to the target class? To tackle this limitation, perturbation-based explanation methods are proposed. These methods perturb the input and aim to find the smallest region, which alone allows a confident classification or prevents a confident classification once being removed (Dabkowski & Gal, 2017; Fong & Vedaldi, 2017) . By applying various ad hoc constraints, these methods improve explainability and maintains faithfulness2 . Nevertheless, most of these methods either focus on distinguishing the supporting features from background and ignore the discrimination on these features, or are only able to predict discriminative scores on a small part of supporting features. To understand the behaviour of a classifier, the discriminative scores over all supporting features are preferred. The comparison of an expected salinecy map to others are shown in Fig. 1(a) 3 . Moreover, the information gap between the classification loss and the constraints in turn increases the difficulty of tuning hyper-parameters. To address these issue, we propose a simple but effective mask predictor. The work is built upon the following observation: a large portion of contributions to each decision are held by only a small fraction of features (Fong & Vedaldi, 2017; Chattopadhay et al., 2018; Du et al., 2018a) . The proposed predictor consists of a mask generator and a distribution controller. The former takes the hidden feature maps in black-box classifiers as inputs, and the latter guides the outputs with expected distributions as relevance scores. We show that, with an easy setting of the involved hyperparameters in the controller, it can directly enforce the relevance scores towards right-skewed distributions (Clauset et al., 2009) . Then, the scores of supporting features becomes more discriminative, which correspond to the small portion of the right tail, and the majority features have low scores and are regarded as unimportant. For illustration, an example of a right-skewed distribution is shown in Fig. 1 (b), and the masks predicted based on the distributions controllers with the right-skewed to the left-skewed are displayed in Fig. 1(c ). Furthermore, we introduce classification losses to optimize mask predictors, which removes all constraints and avoid non-trivial hyper-parameter tuning. We show that such a simplifications will not sacrifice the explainability too much with an ablation study. Finally, we introduce two metrics for comprehensively evaluating relevance scores in terms of faithfulness and explainability, respectively. The main contributions of our work are as follows.

• We present the conception of distribution controllers on relevance scores, and integrate it with a trainable mask generator to improve the discrimination of supporting features. We provide two practical implementations of controllers to enforce scores towards the desired right-skewed distributions, where the involved hyper-parameters can be easily set. • We introduce the classification loss to train the proposed model. It avoids the non-trivial hyper-parameter tuning on ad hoc constraints and also improves the faithfulness of mimicking target black-box classifiers. • We empirically demonstrate the effectiveness of the above innovations. Specifically, we change the setting of distributions and show the controllers can guide the scores towards varying preset distributions. Besides, we perform an ablation study by adding constraints back to analyze their effect on masks. Furthermore, the experiments also demonstrate that our method not only obtains higher quantitative performance for explaining the behavior of black-box classifiers, but also provides discriminative masks for intuitive explanation.

2. RELATED WORK:

This section reviews local explanation methods for DNNs, which target to identify the relevance score of each feature towards a specific decision (Montavon et al., 2018) .

2.1. GRADIENT-BASED LOCAL EXPLANATION:

To obtain relevance scores, gradient-based methods compute the partial derivative of the class probability with respect to the input by using back-propagation (Simonyan et al., 2014) . In general, these methods are advantageous in their high computational efficiency, i.e., using a few forward and backward iterations is sufficient to generate saliency maps. However, these saliency maps based on the naive gradients are visually noisy and hard to be understand. To address this issue, various methods have been proposed. For example, Smooth Grad (Smilkov et al., 2017) addresses the visual noise by introducing noise to inputs repeatedly. Integrated gradient (Fong & Vedaldi, 2017) estimates the global contribution of each feature rather than the local sensitivity. Guided back-prorogation (Springenberg et al., 2014) modifies the gradients of RELU functions by discarding negative values at the back-propagation process. Besides, recent methods proposed to create saliency maps by combining the gradients with the corresponding features. For example, Grad CAM (Selvaraju et al., 2017) and Grad CAM ++ (Chattopadhay et al., 2018) take advantages of high-level feature maps, which makes their saliency maps more clear. Nevertheless, since black-box classifiers are trained without any location information, the object locations in their high-level layers may not always correspond to the locations in raw images, leading to new issues for these explanation methods.

2.2. PERTURBATION-BASED LOCAL EXPLANATION:

The perturbation-based methods first perturb an input according to a given mask, and then observe the new class probability of black-box classifiers. By measuring its difference to the probability of the raw input, the supporting features in the input with the class label can be located (Dabkowski & Gal, 2017) . Let M indicate the expected mask of an image I, where m ij ∈M indicates the relevance score of the pixel within I. Besides perturbing I with mask M, these methods also introduce an alternative background image A to reduce the amount of unwanted evidences:

φ(I, M) = I M + A (1 -M).

(1) Suppose f c indicates the probability of the predicted class c based on the black-box classifier. These method input the perturbed images to the classifier and optimize the mask with ad hoc constraints:

argmin M Ex (M) = argmin M λ av AV(M) + λ tv TV(M) -f c (φ(I, M)) + λ bg f c (φ(I, 1 -M))

(2) where AV aims to minimize the average of all scores and TV enforces the mask to be smooth. The last two terms aim to obtain discriminative scores between supporting pixels and the rest pixels.

To improve the explainability, various novel methods following the above formulation are subsequently proposed. For example, (Du et al., 2018b) regularizes the expected mask with middle-level features and optimizes the mask by reconstructing higher-level feature maps. (Fong & Vedaldi, 2017) introduces a deletion game and reformulates the problem by applying multiple masks stochastically. (Fong et al., 2019) introduces extreme perturbation with an area constraint. However, these methods need to optimize the mask for each image individually, leading to non-negligible time costs.

3. THE PROPOSED METHOD:

In this section, we introduce the details of the proposed framework, as shown in Fig. 1(b) . It consists of an encoder inside the black-box classifier F, a mask generator G, and a distribution controller C. In particular, the generator and controller together compose the mask predictor, which takes the feature maps of the black-box classifier as inputs and predicts the relevance scores for each instance. In the following sections, we first review the mask generator and then introduce the proposed predictor.

3.1. THE REVIEW OF MASK GENERATOR:

We first describe the mask generator G in our framework. Specifically, it contains three bottleneck blocks and follows the U-Net architecture with the black-box classifier (Dabkowski & Gal, 2017) . Each image is input into the classifier for producing feature maps at multiple layers. Then the mask generator upsamples the feature maps of lower resolution using transposed convolutions, and concatenates them with the higher-resolution feature maps. Based on multiple transposed convolution layers, the generator produces a reduced mask M g =(G(F(I)) at a coarse scale and obtain the relevance score on each location. Then, upsampling based on bilinear interpolation is employed to obtain more smooth masks at the image scale. Let M indicate the expected mask. Denote F(I) as the feature maps for the image I. The image is then perturbed with the generated mask with Eq.1, and the generator can be optimized based on the constraints in Eq.2:

argmin G Ex (M), where M = upsample(M g ).

(3) In particular, a simple feature filter is pretrained to perform initial localization with respect to the predicted class. Since it is not the focus of this paper, we leave the detailed description in A.1.

The generator module can produce explanation masks in real time. However, the problem of this method is that, it only tries to distinguish the supporting features from others, without considering the discrimination of scores among these supporting features for better explainability. Besides, balancing the trade-offs between the classification loss and additional constraints, such as the smoothing term in Eq. 2), involves a non-trivial hyper-parameter tuning process. For illustration, an example is shown in Fig. 2(a) , where the masks are obtained from the trained generators under the following hyper-parameter settings: (1) carefully-tuned λs, (2) λ av =0, (3) λ bg =0, (4) λ tv =0, (5) λ bg =λ tv =0. As we observed, each term in Eq. 2 brings a significant effect on final masks. Of note, although λ bg =0 can release the masks to obtain discriminative scores on a small part of supporting pixels, it reduces the explainability on the others. Besides, it increases the sensitiveness of λ av on masks. The reason is that by enforcing f c (φ(I,1-M)) to low values, masks can highlight the majority of supporting pixels even when λ av is slightly larger. Once it is removed with λ bg =0, the size of high scores is totally controlled by λ av . More details are discussed in Sec. 4.4.

3.2. THE MASK PREDICTOR:

To address the above issues, we first present a distribution controller after the above mask generator to guide the relevance scores. Then, we introduce classification losses for optimizing the predictor without ad hoc constraints. The details of two technical innovations are presented below.

3.2.1. THE CONTROLLER ON DISTRIBUTION:

To produce discriminative scores on supporting features in M, we introduce the conception of distribution controllers C, which is used to guide the relevance scores towards desired distributions. Suppose M g is the output of the generator on I, and M d denotes the output of C. The output of distribution controller is expressed as

M d = C(M g ).

(4) Based on the above transform, we expect to encode the distribution inside the mask predictor rather than bringing the formulation an additional constraint. Besides, we follow (Dabkowski & Gal, 2017; Du et al., 2018b) to upsample M d with interpolation, aiming to improve the smoothness of masks.

The principles of controllers. Now we investigate two principles for designing the controller C.

Principle 1. We expect that the output scores of C approach a right-skewed distribution for each instance. The motivation behind is that a large portion of contributions to each decision are supposed to be held only by a small fraction of supporting features with varying importance. These supporting features, whose information should be preserved by the mask, are assigned with higher but discriminative scores.

Principle 2. We impose the monotonic mapping from the distribution controller's input M g to its output M d . The purpose is to enhance the discriminative ability of the coarse mask M g but without changing its ranking of relevance scores.

For better illustration, two examples are shown in Fig. 1 (a) and Fig. 2(c ). In the first example, with the expected distributions of controllers varying from the right-skewed to the left-skewed, the dis-crimination of supporting features is remarkably reduced. Besides, the proportion of features that are highlighted with high scores is nearly positive correlation to the area at the right part of preset distributions. In the second example, a non-monotonic transform g(x) = x 2 is used in controllers, which results the scores in the deviance from expected meanings. That is, a higher score implies a larger contribution.

The implementation of controllers. Following the above principles, we introduce two piratical implementation of the controllers. Since a sum of even two independent random variables is more Gaussian than the original variables (Hyvärinen & Oja, 2000; Ioffe & Szegedy, 2015) , we first assume that the input scores of controllers (the outputs of generators) are nearly Gaussian for convenience. Later we show that the proposed controller built upon this assumption can also benefit other distributions towards the right-skewed.

From the Gaussian to the normal. In order to obtain the desired distributions, we first introduce instance normalization (Ulyanov et al., 2016) to transform Gaussian distributions towards normal distributions. The goal of this step is to shift scores around two sides of 0.

Specifically, let m g ij ∈M g indicates the input entry at the location (i, j) of C, and z ij ∈Z denotes the expected variable with a normal distribution. The mapping can be expressed as

EQUATION

where the expectation E[•] and variance Var[•] are computed over the entries of each M g .

From the normal to the right-skewed. Now we further transform the above scores towards a rightskewed distribution monotonically. To do this, we introduce two simple transformers, including a basic one without any hyper-parameter, and a customized one with easy-to-set hyper-parameters.

Basic Transformer (BT): The straightforward way of transforming a normal distribution into a rightskewed distributions is cropping variables with Rectified Linear Unit (ReLU) (Nair & Hinton, 2010) . Specifically, the expected output of a distribution controller m d ij can be obtained as

m g ij = ReLU(z ij ) = max(0, z ij ). (6)

The benefit of BT is to enable a monotonic transformation without any hyper-parameter. However, BT is lack of the potential ability to handle other distributions, such as the left-skewed inputs.

Customized Transformer (CT): To address the issues of BT, we introduce an alternative with easyto-set hyper-parameters. The goal is to produce scores for all features with a right tail in (0,1). Specifically, we first transform the normal distribution towards an uniform distribution based on sigmoid functions, and then change the skewness of the distribution based on power functions:

EQUATION

where η is used to approach the uniform distribution (Waissi & Rossin, 1996) , and h determines the skewness of the obtain distribution. Based on the probability density transformation (Forbes et al., 2011) , we can easily set these hyper-parameters based on the intuitive geometric properties of their probability density functions. For example, with η=1.5 and h=2.5, we can obtain the scores with the probability density function in Fig. 1(b ). Owing to the space limitation, we leave the detailed discussion to A.2. Note that other transformers with monotonicity can also be considered.

The affect on other distributions. Next, we release the Gaussian assumption and consider the effect of the preset controller on other distributions. Specifically, we introduce the CT with η=1.5 and h=2.5 on these distributions. The original (in gray) and transformed distributions (in blue) are displayed in Figs. 3(a-d ). As we can see, although CT may not transform them to the right-skewed perfectly, it can still shift a part of scores to low values and make scores away from the left-skewed.

The post hoc tuning of distributions. In practice, constraining different images with a same distribution may lead to redundant scores on unimportant pixels, especially when the supporting pixels only take a small part of the image. For this, we present a simple trick to perform a post hoc tuning.

Given a predicted mask, which ranks all image pixels with respect to their scores for the predicted class. We first iteratively add ranked pixels from each image and measure the resulting probability q i for i pixels. Next, we modify these probabilities with q i =max q j≤i to ensure the monotonicity. Since the pixel i is not likely to be the supporting one if q i already approaches the maximum q max =max i q i , we post hoc tune the relevance score as r i =r i ×v i , where v i =(q max -q i )+eps, and r i is original predicted score of the i-th pixel. An example is shown in Fig. 3 (e), where post hoc tuning is performed on the masks that obtained from BT (the top two) and CT (the bottom two).

Of note, benefiting from the enforced monotonicity, the ranking of relevance scores can be remained.

Besides, we uniformly sample a limited number pixels for estimating the probabilities and infer the remaining with linear interpolation.

3.2.2. OPTIMIZATION WITH CLASSIFICATION LOSS:

In this section, we introduce classification loss to optimize the above mask predictor. Denote Cl as the classification loss used for training the black-box classifier f , the proposed predictor can be optimized without the non-trivial hyper-parameter tuning as

EQUATION

Since the predictor is formulated to find the region that maximizes the target classification under the expected distribution, it can also better mimics the black-box classifier.

It is observed that when the cross entropy loss is used for training the original classifier f , Eq.8 is similar to removing all ad hoc constraints in Eq.2.

Thus, one may argue that the abandon of these constraints may remarkably scarify the quality of masks. For this, we provide an example of the ablation study by adding all constraints back and then display the masks from the trained predictors in Fig. 2 (b) under the following hyper-parameter settings: (1) carefully-tuned λs, (2) λ av =0, (3) λ av =λ bg =0, (4) λ av =λ tv =0, (5) λ av =λ bg =λ tv =0. In general, we still obtain satisfying masks without any constraint. Detailed analysis is left to Sec. 4.4.

4. EXPERIMENTS:

This section investigates the performance of the proposed method. We firstly evaluate the faithfulness and explainability with quantitative metrics. Then we perform the visualization of masks for further comparison. Finally, we introduce the case study to empirically demonstrate the effectiveness of two proposed innovations, respectively.

4.1. SETUP:

In experiments, to demonstrate the broad applicability, we applied our method to CNNs with three types of structures. Besides, we compare Ours(BT) and Ours(CT) with nine recently proposed explanation methods. The experiments are performed on a real-world dataset, ImageNet. A brief introduction is given below, and details can be found in A.3.

Three types of CNNs. We use ResNet50 (He et al., 2016) , VGG19 (Simonyan & Zisserman, 2014) , and GoogleNet (Szegedy et al., 2015) as black-box classifiers to show the broad applicability. We load the parameters from traditional CNNs that were pre-trained using 1.2M ImageNet images.

Nine compared methods. The following methods are used for comparison: (1) Mask Generator (MGnet) (Dabkowski & Gal, 2017) , (2) Meaningful Perturbation (MPert) (Fong & Vedaldi, 2017) , (3) Grad CAM (GCAM) (Selvaraju et al., 2017) , (4) Grad CAM++ (GCAM++) (Chattopadhay et al., 2018) , (5) Feature Inversion (FInv) (Du et al., 2018b) , (6) Extreme Perturbation (XPert) (Fong et al., 2019) , (7) Vanilla Gradient (VGrad) (Simonyan et al., 2014) , (8) Smoothness Gradient (SM-Grad) (Smilkov et al., 2017) , (9) Integrated Gradient (ITGrad) (Sundararajan et al., 2017) . In par-ticular, all perturbation-based methods apply the same strategy to ours for adding noise. For the hyper-parameters in these methods, we follow the setting in their papers for a fair comparison.

Implementation details. We build the mask generator with three bottleneck blocks, which takes 7×7 feature maps as the low-level input, and predicts the mask at the 56×56 in size. We use a twostage scheme to train the mask predictor. We first train the feature filter based on 250,000 images sampled from the training set, and then optimize other parts of the mask predictor with the batch size of 64. Of note, no ground truth is introduced and only the outputs of the classifier are utilized. We use Adam for 10 epochs with the initialized learning rate of 10 -2 . We apply step decay, and reduce the learning rate by half every three epochs. In addition, during training stage, 50% of cases the image A is the Gaussian blurred version of I with a variance of 10. The remainder of cases, A is set to a random colour image with the addition of a Gaussian noise.

Quantitative metrics. Evaluating the quality of masks based on a heuristical segmentation will reduce the fairness of comparisons. It is understandable that the segmentation will be significantly affected by thresholds (Fong & Vedaldi, 2017) . To address this issue, we introduce two generalized metrics to evaluate the comprehensive quality in terms of both the faithfulness and the explainability.

Faithfulness. The explanation is expected to accurately replicate the model's behaviour. Here we rely on the smallest sufficient region of the image that alone allows a confident and consistent classification. Since it is subjective to decide how much confidence is preferred for a specific decision, we extend the evaluation into a general case by taking advantage of ranked pixels.

Specifically, we first estimate the class probability of a clean image as q O . Then we iteratively add i ranked pixels with decreasing scores and measure the resulting probability q i of the target class, until the fully-perturbed images q F . Then, a preserving curve can be generated. The increased area under the curve with the intervals of ∆ is used as a measure of faithfulness, calculated as

EQUATION

An extra discussion between M F and perturbation-based methods are provided in A.4.

Explainability. For quantitative evaluation of explainability, we generally use the extra information such as bounding boxes. For example, in the pointing game, the maximum point is first extracted from each generated mask. Then, according to whether the maximum point falls in one of the ground truth bounding boxes or not, a hit N Hits or a miss N Misses is counted. The localization accuracy of the pointing game for each object category is defined as:

EQUATION

)

This process is repeated for all categories and the results are averaged as the final accuracy. However, M PG only unitizes one pixel for evaluation, leading to a large bias. To address this issue, weaklysupervised object localization considers a boarder pixels by calculating the intersection over union between binary masks and images. Nevertheless, it still faces the issue of the choice of thresholds.

Thus, we introduce a new metric by regarding relevance scores as the results of retrieval tasks. Specifically, we denote the precision as the fraction of the pixels retrieved within bounding boxes.

Recall denotes the fraction of the within-bounding-box pixels that are successfully retrieved. By computing a precision P i and recall R i at each position in the ranked scores of pixels, we can evaluate the explainability of scores by the area under the precision-recall curve:

EQUATION

)

4.2. QUANTITATIVE EVALUATION:

On the faithfulness with M F . To evaluate the faithfulness of the proposed predictor, we calculate the mean metric M F based on 10,000 sampled images from the validation set. We set ∆= 1 32 as the interval. Besides, we simply add a smoothed mask over the original one with a tiny weight, introducing the ranking for pixels with zero scores. We perturb images with their Gaussian blurred version based on their masks and estimate the average faithfulness. In order to avoid outliers, we introduce min-max normalization on M F s over all methods for each image. Of note, although post hoc tuning remains the ranking of pixels, it is not performed for avoiding the ambiguity. The results of average M F are displayed in Fig. 4 (left). Based on the results, the following observations can be obtained. Firstly, VGrad, SMGrad, and ITGrad are generally worse than others with a large gap. It is understandable that, these three methods only search sensitive pixels with gradients, the pixels with high scores will discretely appear in each image. As a result, these methods become harder to gather the sufficient supporting information in a local receptive field and reach a high class probability. Secondly, GCAM obtains comparable performance to GCAM++. Besides, FInv that regularizes masks by utilizing intermediate layers outperforms MPert. It suggests that, without using the lowest resolution feature maps, an effective regularization is of importance for individually optimizing the mask of each instance. Thirdly, our methods outperform most compared methods, and obtain much higher performance than MGnet. Consider that the three methods apply the same architecture of mask generators. Thus, the results demonstrate that guiding the relevance scores towards right-skewed distributions enjoys a better ranking of supporting features.

On the explainability with M I . To reveal the explainability, we first introduce M I to evaluate the performance of object location. For this, we resize and crop bounding boxes to the size of 224×224, leading to the same size of test images. The experiments are performed on 10,000 validation images with bounding box annotations.

The results of average M I are displayed in Fig. 4 (middle). From these results, we have the following observations. Firstly, GCAM++ consistently obtained much better performance than GCAM. It is understandable that the former is able to detect multiple objects in the image and assign them high relevance scores. Secondly, by replacing all constraints with a simple distribution controller, Ours(CT) outperforms MGnet with a small gap, and both of them enjoy better performance than most of other methods. The reason is that, benefiting from the training with large-scale images, their predicted high relevance scores focus on objects more robustly. Thirdly, the last three gradientbased methods perform worse than others. The possible reason is that, gradients are insensitive to the smooth supporting regions, which makes these regions ignored and reduces the performances.

On the explainability with M PG . In this experiment, we show that our methods still bring comparable performances in pointing games. The average results of all compared methods on pointing games are shown in Fig. 4 (right). From the results of this figure, the following observations can be obtained. Firstly, SMGrad and ITGrad now obtain comparable or even better performance than the first eight methods and outperform VGrad. The main reason is that, this metric only considers the localization precision of the top pixel rather than a group of pixels. By removing noise in gradients to some degree, these two methods are able to locate the most important pixel. Secondly, although our methods remove all constraints, e.g, λ bg =0, and λ tv =0, they still obtain the comparable accuracy of locating the most important pixel. One possible reason is that, by optimizing the predictor with training data, a large number of noisy features can be removed, leading to the robust estimation of scores. Thirdly, MGnet enjoys the best performance. It means that, by enforcing the background towards a low probability, the high scores on the background are generally prevented.

4.3. EXPLAINABILITY WITH VISUALIZATION:

Comparison to baseline methods. We qualitatively compare the saliency maps produced using our methods with those produced by other baselines. The red and blue denotes the high and low score, respectively. In this experiments, no post hoc tuning is used for the proposed methods. We randomly sample images from ImageNet and show their masks obtained with ResNet50 in Fig. 5 From these results, we have the following observations. Firstly, the gradient-based methods generally results in more high scores outside the objects. Secondly, although the high-level feature maps are used to build saliency maps, GCAM and GCAM++ may still miss the supporting pixels of objects, such as the last second row in Fig. 5 (left). Thirdly, MGnet obtains high scores on the supporting features. However, it only assigns them similar relevance scores. Finally, compared with other methods, with an easy setting of hyper-parameters, Ours(CT) not only predicts higher scores on these pixels than others but also builds discriminative scores over these pixels. It shows the effectiveness of guiding scores under right-skewed distributions.

The effect of post hoc tuning. In this experiment, we aim to investigate the effectiveness of post hoc tuning to improve the distribution of relevance scores. Fig.5(right) shows same examples of our masks with ResNet50 before and after the tuning operation. The masks of MGnet are also displayed for comparison. More examples of this task can be found in Fig. 13 in A .5.

From the figure, it is observed that compared with MGnet, our predicted masks sacrifice the difference of scores between the supporting features and others to some degree. However, owing to the better ranking of scores over pixels (which is empirically demonstrated by the metric M F ), we can easily utilize the post hoc tuning to remove the noise and highlight the supporting pixels.

4.4. PERFORMANCE OF THE INNOVATIONS IN OUR PREDICTOR:

On distribution controller: the effect of distributions. To show that the distributions of output masks are benefited from the proposed controller, we study the masks from the predictors optimized under varying preset distributions. Specifically, we change the hyper-parameters of controllers with {(η,h)}={(0.5,2.5),(1.5,2.5),(1.5,1.5),(2.5,2.5),(1.5,0.5)}. Then we train mask predictors with these controllers. The resulted masks with the corresponding preset distributions are shown in Fig. 6 (left).

As we observed, although it is hard to expect that all masks obtain scores with the same distribution to the expected one, the controllers enforce them towards these distributions. Besides, we perform the transformer on the obtained masks of MGnet in a post hoc manner, which fails to build discrim- inative scores on supporting features. The reason is understandable that the scores are already be stacked at the values of 0 and 1.

On optimization with classification loss: the effect of regularization. To evaluate the impact of our simplified objective function, we introduce an ablation study to analyze how the regularization terms affect our mask predictor. We add all ad hoc constraints back to the formulation and train the mask predictor using the following hyper-parameter settings: (1) carefully-tuned λs, (2) λ av =0, (3) λ av =λ bg =0, (4) λ av =λ tv =0, (5) λ av =λ bg =λ tv =0. For comparison, an ablation study is also perform on MGnet with the setting of (1) carefully-tuned λs, (2) λ av =0, (3) λ bg =0, (4) λ tv =0, (5) λ bg =λ tv =0. The masks from the trained models with varying settings are displayed in Fig.6(right) .

By adding all ad hoc constraints back, the masks of Ours(CT) can be improved to some degree. However, we observed that λ av =0 has few effect on our predictor. In addition, λ bg =0 will increase the noise around the boundary owing to the smoothness constraint, and λ tv =0 causes the holes in the masks. Nevertheless, by further removing these two terms, these shortcomings can be alleviated and the final masks of Ours(CT) still remain satisfying quality. On the contrary, besides λ tv =0, all terms result in remarkable impacts on the masks of MGnet. Of note, although λ bg =0 can release the mask towards discriminative scores, it would reduce the discrimination on the majority of supporting features, such as the second, fourth and fifth rows. Besides, it increases the sensitiveness of λ av on masks. For example, while λ bg is carefully set, λ av varying within (2,12) consistently leads to acceptable masks for ResNet50. Once λ av =0, the number of high scores is totally controlled by λ av and the quality of masks is only acceptable with λ av ∈(2,4). In short, benefiting from the distribution controller, Ours(CT) is insensitive to the regularization terms and can predict satisfying masks without ad hoc constraints.

5. CONCLUSION AND FURTHER WORK:

This paper presents a simple but effective mask predictor to provide local explanations for blackbox classifiers. Specifically, we introduce the conception of distribution controller on relevance scores, and integrate it with a trainable mask generator to directly guide the relevance scores. In addition, the proposed predictor is optimized under the classification loss, aiming to better mimic the classifiers. Based on the proposed innovations, our method enables discriminative scores over supporting features and also facilitates the setting of involved hyper-parameters. The experimental results demonstrate that our method outperforms others in terms of faithfulness and explainability.

There are some aspects needing further investigations. Firstly, since the objective functions are formulated the same to those of classifiers, it is natural to train the classifiers and mask predictors at the same time. As a result, we can provide users a pair of classifier and explainer. Secondly, although this paper provides an intuitive comparison of the transformed distributions for setting hyper-parameters, a quantitative analysis on the proportion of features at the tail could be studied. We fix h=1 and observe the effect of η. The corresponding probability density function is displayed in Fig. 7(a) . By changing η within (0.5,2.5), the density functions of p(m d ) approximately change from the concave to the convex for m d ∈(0,1). In particular, η=1.5 approximately leads p(m d ) to an uniform distribution. Considering 1.5∼0.9 √ π, it is also consistent to the sigmoid approximation of the cumulative probabilities of the standard normal distribution (Waissi & Rossin, 1996) .

Then we set η={0.5,1.5,2.5} and observe the effect of h. Three probability density functions are displayed in Figs. 7(b-d ). As we can see, p(m d ) with η=0.5 leads to extremely low probabilities for m d >0.5, which will potentially waste this range of scores for highlighting supporting features. Besides, owing to the strong convexity in the first step of the transformer, p(m d ) with η=2.5 continues this undesired property. On the contrary, benefiting from an approximated uniform distribution with η=1.5, the distribution of p(m d ) in Fig. 7(d ) is modified from the trend of the left-skewed to the right-skewed with a light tail over (0, 1). Since a proportion of features corresponding to the area at the tail will be highlighted with higher scores, we fix η=1.5 and h=2.5 to obtain discriminative scores on supporting features.

Proof. According to the probability density transformation Forbes et al. (2011) , the transformed probability density function can be obtained based on that of the original variable:

EQUATION

where p z (•) is the probability density function of the original variable z, and g -1 z (m) is the inverse function of m on z. Since m=g z (z)=( 1 1+exp(-ηz) )

h ∈(0,1) in Eq.7, we obtain

EQUATION

we set three low-to-high-resolutions blocks in mask generators with the channels of {256,128,64}, namely, one quarter channels of the classifier; for VGG19 and GoogleNet that contain the layers with {64,128,256,512,512} and {64,192,480,832 ,1024} channels, we half the channels in mask generators, namely, {256,128,64} and {416,240,96} for these blocks, respectively.

A.4 THE ANALYSIS ON THE RELATIONSHIP BETWEEN THE PROPOSED METRIC AND THE

PERTURBATION-BASED METHODS:

We provide a new view to discuss the relationship between the proposed metric and wildly used perturbation-based methods with the mask involved. Recall that in most perturbation-based methods, we aim to maximize the probability of the target class of the perturbed image:

EQUATION

where the pixels with higher scores in masks are supposed to be more important. By decomposing the above mask into multiple ones with one-hot coded mask M i , Eq.19 is equal to

EQUATION

where α i denotes the relevance score with α i >α i+1 . Suppose β i indicate some positive weights and denote α i = N j=i β j , the above equation can be further transformed into

EQUATION

Recall the area under the probability vs. the number of pixels, which can be formulated as 

We observe that Eq.21 tends to be a linear approximation of Eq.22. Specifically, two of them become equal when f c (•) and φ(•) are linear. According to Eq.22, the larger values on the former β i means the users pay more attention to the top features. According to Eq.21, it also implies the high scores should be much larger than others and highlight supporting features. Thus, to evaluate the effectiveness of perturbation-based methods, the proposed metric tends to be more convincing.

A.5 SUPPLEMENTARY RESULTS ON VISUALIZATION EVALUATION:

In this section, we provide the supplementary results on visualization evaluation.

Specifically, in Figs.8-12, we display the predicted masks obtained from different methods with ResNet50, VGG19, and GoogleNet. By compared the proposed methods with others in these figures, we observe that our methods are generally able to distinguish the objects from the backgrounds, and also obtain discriminative scores at different parts of objects. It thus provides better explanations for the behaviours of black-box classifiers.

Besides, to further demonstrate the effect of post hoc tuning, we show more masks before and after the tuning operation in Fig. 13 . As we can see, benefiting from the effective ranking of supporting pixels, we can avoid the noise outside of the objects and further improve the masks.

:

In this paper, relevance scores indicate the contributions of features to a specific decision. A high score implies a higher contribution. Besides, we do not discriminate saliency maps and masks, as both indicate the permutation of relevance scores of each input.

Explainability quantifies how easy it is to understand and reason about the explanation; faithfulness estimates the fidelity between the explanation and the decision behaviour of black-box classifiers.

3 Besides highlighting the pixels within the dog for the images of dogs, the relevance scores should also be discriminative at various parts of the dog, such as higher scores on the head than legs.

Figures and Tables:

Figure 1: (a) An example of the comparison of the expected saliency map (the last one) to others. (b) An example of right-skewed distributions. (c) The illustration of the benefits of using right-skewed distributions for explanation. (d) The framework of our mask predictor, where a distribution controller is introduced right after the mask generator. In particular, pdf stands for the probability density function.

Figure 2: The examples of the ablation studies on (a) (Dabkowski & Gal, 2017) and (b) the proposed method.(c) The negative effect of non-monotonic transformers.

Figure 3: (a-d). The effect of CT on different distributions. (e) The effect of post hoc tuning on masks.

Figure 4: The quantitative results of eleven methods on three CNNs. (left) Performance on MF , (middle) Performance on MI, (right) Performance on MPG. Detailed values are listed in the above table.

Figure 5: (left). Masks of different methods with ResNet50. (right). The effect of post hoc tuning.

(left). More examples from other CNNs are displayed in Figs.8-12 in A.5.

Figure 6: (left) The effect of preset distributions (1st row) on the predicted masks (2nd-8th rows). (right) The ablation study on the regularizations of different methods.

Figure 7: The probability density functions with the different settings of hyper-parameters.

j ))) = β 0 f c (φ(I, M 0 ))) + β 1 f c (φ(I, M 0 + M 1 )) + ....

Peer review (iclr2020):
Paper Summary: This paper proposed a method to produce instance-wise saliency map for image classification tasks. The proposed method develop a U-net-based generator for saliency mask, where the important modifications are (1) a skewness-inducing activation function for mask generation (i.e. controller), which is either a ReLU function or a scaled sigmoid function raised to a certain power. Authors argue that the proposed activation function leads to output saliency score to have right-skew distribution, which leads to more distinguished saliency map. (2) a smoothness-inducing mechanism where the saliency mask is generated at coarser scale then up-sampled with a bilinear operation so the generated map is smooth. Finally, the training is performed by minimizing solely on the cross-entropy loss with respect to the original model predictions, which is expected to improve faithfulness in mimicking the original black-box classifiers. I*m leaning toward rejecting this paper in its current form. While I think this paper proposed an interesting strategy in improving the faithfulness of the explainer (i.e. training on the cross-entropy loss with respect to original classifier), the rest of the two modification either already exists in the literature (generate mapping at coarser resolution is an idea from Dabkowski & Gal (2017) / Du (2018) as pointed out by the author) or suffers potential technical issues that can benefit from further methodology improvement/empirical justification (please see Major Comments). Furthermore, empirical experiments seem to suggest that the proposed method sometimes fail to produce more smooth and more discriminative maps compared to its MGnet baseline (e.g. Figure 5, top right row), calling into the question of whether training solely on a cross-entropy loss is enough to ensure the generated saliency map is of high-quality. I still like the overall idea of this paper, and I think this work can be made more rigorous and informative by including a careful ablation study about the loss/gain of different regularization terms and loss functions (e.g. the effect of TV(M) term on smoothness, the effect of f(phi(I, 1-M)) term on discriminativeness, and the trade-off between cross-entropy v.s. the original negative log likelihood loss as used in Dabkowski & Gal (2017)). Major Comments: (1) Custom Transformer The derivation of the new activation function assumes that the distribution of the hidden-layer output is strictly Gaussian (i.e. unimodal, symmetric, and with sufficiently light tail), and author proposed to build this Gaussian distribution from raw hidden-unit output through standardization (Section 3.2.2). However, this approach may not be valid in practice. This is because there is no guarantee that the distribution of is strictly unimodal and symmetric. If the output is skewed/multimodal in anyway, the normalization operation won*t produce a Gaussian distribution, since a linear transformation (i.e. normalization) of a skewed distribution is still a skewed distribution. As a result, I am worried that the hyperparameter derived in Section 3.2.3 may not always lead to right-skewed distribution as expected. If author would like to ensure the outcome distribution is right-skewed, it may be good to tune these activation function parameters for each instance so the empirical distribution of satisfy certain skewness criteria (e.g., maybe tune the parameter in a way such that the pearson*s coefficient of skewness of the empirical distribution is sufficiently small). In addition, I find the author*s claim about the benefit of right-skewed distribution (i.e. producing more discriminative maps) not yet well supported by the empirical evidence. For example, in Figure 4 the produced maps in the first three rows are relatively fuzzy, and the difference between the irrelevant features and the important features does not seem to be very high. The situation is only improved after the instance-wise finetuning. I think there are two possible reasons, either (a) the right-skewness of the actual distribution of the saliency map score in those images were not guaranteed (as mentioned earlier, I believe the current approach cannot guarantee the pre-activation units are Gaussian distributed, therefore the actual skewness of the post-activation values may vary depending on the input). (b) the right-skewness in the outcome distribution alone is not enough to guarantee good discrimininative behavior. To clarify this (and provide more evidence for author*s claim that skewness <-> better discrimination), author can consider visualizing the saliency map with varying degree of skewness (you can measure the skewness of the empirical distribution of the saliency-map score using pearson*s coefficient of skewness), and show that the *discriminativeness* and skewness of the empirical distribution are correlated. (2) Smoothness of the produced map In the original MGnet work, the authors used two mechanisms to guarantee smoothness of the saliency map: (1) generate map at coarser level and (2) TV penalty on loss function. In this work author used only (1). As a result, the produced map in some cases (e.g. Figure 5, top right row for BF After, also Figure 7, left fifth row for BF after) appear jagged and not smooth. I*m wondering if adding TV penalty back would help somewhat. Again, a careful ablation study would be beneficial in clarifying this. Minor Comments (1) There*s some minor technical issue in the description for Equation (6). Assuming is Gaussian, it is AND sigmoid() COMBINED to *approach the uniform distribution*, it is important to mention the sigmoid function since the uniform transformation used here is based on the Probability Integral Transform, (i.e. if a random variable follows Gaussian distribution with CDF , then , i.e. you are transforming a Gaussian random variable using Gaussian CDF to get uniform distribution), and equation (6) is using AND sigmoid() combined together to approximate this Gaussian CDF [1]. (2) Equation (14) on page 13. There should be an term within the exponential function. Reference: [1] Gary R.Waissi, Donald F.Rossin (1996) A sigmoid approximation of the standard normal integral.