Title: SALIENT EXPLANATION FOR FINE-GRAINED CLASSIFICATION

Abstract: Explaining the prediction of deep models has gained increasing attention to increase its applicability, even spreading it to life-affecting decisions. However there has been no attempt to pinpoint only the most discriminative features contributing specifically to separating different classes in a fine-grained classification task. This paper introduces a novel notion of salient explanation and proposes a simple yet effective salient explanation method called Gaussian light and shadow (GLAS), which estimates the spatial impact of deep models by the feature perturbation inspired by light and shadow in nature. GLAS provides a useful coarseto-fine control benefiting from scalability of Gaussian mask. We also devised the ability to identify multiple instances through recursive GLAS. We prove the effectiveness of GLAS for fine-grained classification using the fine-grained classification dataset. To show the general applicability, we also illustrate that GLAS has state-of-the-art performance at high speed (about 0.5 sec per 224×224 image) via the ImageNet Large Scale Visual Recognition Challenge.

1. INTRODUCTION:

Over the last several years, convolutional neural networks (CNNs) (LuCun et al., 2015) have achieved superior performance in various computer vision tasks, including image classification (He et al., 2016; Shi et al., 2018) , object detection (Oh et al., 2017; Zhou et al., 2016) , semantic segmentation (Pathak et al., 2015) , and image captioning (Xu et al., 2015) . Despite these dramatic advances, the opacity of CNNs makes it difficult to understand why they reach particular decisions, limiting the ability to widen their application to various fields.

In general, the visual interpretation of deep learning models is understood as estimating the impact of a particular neuron activation related to a given input instance. In white-box approach, architectural modification of the classification model (Bach et al., 2015; Dong et al., 2017; Mahendran & Vedaldi, 2016; Selvaraju et al., 2017; Simonyan et al., 2013; Springenberg et al., 2014; Zhou et al., 2016; Zeiler & Fergus, 2013) or access to specific layers (Bach et al., 2015; Selvaraju et al., 2017; Zhang et al., 2016) is inevitable (Petsiuk et al., 2018) , resulting in severe limitation of application. In contrast, the black-box approach (Seo et al., 2018; Petsiuk et al., 2018; Ribeiro et al., 2016; Tian & Cai, 2017; Zeiler & Fergus, 2013; Zintgraf et al., 2017; Fong & Vedaldi, 2017) aims to be inherently model agnostic. Its main concerns are how to perturb an input image and draw the model's response on the perturbed instance to the final heat map. For example, the Randomized Input Sampling for Explanation (RISE) method (Petsiuk et al., 2018) perturbed an image with a randomised mask to measure the importance of pixels and then linearly fused all importance from several thousand masks.

The conventional black-box methods employed unnatural and fragile perturbation schemes such as single colour out (Seo et al., 2018; Petsiuk et al., 2018; Ribeiro et al., 2016; Zeiler & Fergus, 2013) , random noise (Tian & Cai, 2017; Zintgraf et al., 2017; Fong & Vedaldi, 2017) and smoothing (Fong & Vedaldi, 2017) . These perturbation schemes have several limitations. First, they are deficient in pinpointing only the most discriminative, i.e., salient features that are essential for the fine-grained classification tasks where the between-class shape similarity is very high; for example, pinpointing only the red face of Red-faced Cormorant in the bird classification task in Figure 1 is crucial for explaining why a deep learning model classifies the image as Red-faced Cormorant. Second, the conventional perturbation schemes highly suffer from local noise and, thus, fuse maps from a con-Figure 1 : Salient explanation. As the value of decreases, the heat map concentrates more and more on the most discriminative part of the relevant objects, the bird's face in this case. The salient explanation is very important for a variety of tasks such as the fine-grained classification and biomarker discovery in medical image. Note that the salient explanation is possible due to our idea of adopting Gaussian mask.

siderable number of perturbations for a reliable explanation. This is the main cause of slowness with conventional black-box methods.

Inspired by the lighting and shadowing phenomena in nature, we propose a simple yet effective black-box method, called Gaussian light and shadow (GLAS), which simulates feature perturbation as the presence or the absence of light at the pixel level of an image. The primary idea of GLAS is to perturb an input image by the Gaussian mask (light) and inverse-Gaussian mask (shadow) and, then, record the responses of the perturbed images. GLAS uses a simple grid search; once completed over the entire image, the response maps are fused to construct the final heat map. The fusion mimics the Gaussian mixture. The proposed method has several advantages compared with other black-box methods (Petsiuk et al., 2018; Zintgraf et al., 2017; Fong & Vedaldi, 2017) . First, GLAS provides scalability of explanation that we can achieve by adjusting the variance parameter of the Gaussian mask. The scalability makes it possible to pinpoint clues for the salient explanation, which is not feasible with the nonparameterized approaches (Mahendran & Vedaldi, 2016; Zhou et al., 2016; Zeiler & Fergus, 2013; Petsiuk et al., 2018; Zintgraf et al., 2017; Fong & Vedaldi, 2017) . The salient explanation is valid for explaining fine-grained classifications, such as classifying bird species in a CUB200 dataset, involving large between-class similarity and significant within-class variance. Figure 1 shows an image of the Red-faced cormorant species that we can discriminate by identifying the face color. It illustrates that GLAS adjusts its gaze from the body to the red face as the scale parameter decreases and finally pinpoints the red area around the eye. Second, our pixel-wise multiplication operation with the Gaussian mask at a specific search point simulates the gradual dimming effect as going farther from the center. We argue that because of this characteristic, a significantly reduced number of perturbations is sufficient. GLAS can process an image much faster than conventional methods.

To summarize, the contributions of this paper are as follows: We introduce a novel notion of salient explanation which is critical in explaining the fine-grained classification tasks. We propose a simple yet efficient black-box method, GLAS, which provides an easy way to perturb an input image based on Gaussian lighting and shadowing. (1) GLAS is fast because of the smoothly varying shape of the Gaussian mask, which generates a visual explanation up to one order of magnitude faster than other black-box methods. (3) We show the broad applicability of GLAS to various other tasks: object localization and visual captioning. Quantitative comparisons show that GRAS is superior to conventional methods.

2. RELATED WORKS:

The white-box approach heavily uses the network's internal information, such as gradients or feature maps of specific layers. A gradient can indicate how much a small change in a pixel influences the class output (Springenberg et al., 2014) . For example, Simonyan et al. (2013) proposed the gradientbased model, which directly mapped saliency values to the original space. Additionally, Zeiler & Fergus (2013) proposed a deconvolution method. In the method, the forward signal is reversed at a neuron and backpropagated to the input space. The study (Bach et al., 2015) proposed the layer-wise relevance propagation method, in which the prediction in the output layer is decomposed into pixelwise relevance values and backpropagated until it satisfies the conservation rule. Samek et al. (2016) emphasized the importance of quantitative evaluation and provided a rigorous comparison of the previously mentioned methods; these approaches are extensively reviewed in Samek et al. (2016) . The visual feature maps provide important clues for the explanation, which some techniques exploit. The technique (Zhou et al., 2016) called class activation mapping (CAM) is accomplished by weighted fusion of visual feature maps and requires the modification of CNN architecture, replacing the fully connected layer with the global average pooling. Grad-CAM (Selvaraju et al., 2017) , an extended version of CAM, is applicable to a broader range of CNNs. The previously mentioned techniques modify the model's internal operations or rely on the model's internal values; thus, they are model dependent.

The black-box approach measures the response change of the base model when the input instance is spatially perturbed, and this change can be regarded as the significance of the classifier's decision. The study (Robnik & Kononenko, 2008) simulates feature perturbation based on marginal probability, and several studies have extended and improved this method (Tian & Cai, 2017; Zintgraf et al., 2017) . For the CNN-based architectures, the study (Zintgraf et al., 2017) proposed the conditional sampling approach. The method considers that a given pixel value highly depends on neighboring pixels and that multivariate analysis excludes a rectangular region rather than a single pixel. Because a pixel-wise perturbation method such as random noise (Zintgraf et al., 2017; Fong & Vedaldi, 2017) was considered, pixels are highly vulnerable to adversarial attack.

Several techniques (Seo et al., 2018; Ribeiro et al., 2016; Tian & Cai, 2017) aim at region-wise perturbation approaches, rather than using pixels. A study (Tian & Cai, 2017) improved the conditional sampling method using the superpixel algorithm, making it more robust from local noise than Zintgraf's method. Additionally, the superpixel segmentation technique was used in existing methods (Seo et al., 2018; Ribeiro et al., 2016; Tian & Cai, 2017) . In these methods, high-level segments, rather than pixels (Zintgraf et al., 2017) or oversegmented regions (Tian & Cai, 2017) , are used to perturb the feature of instance, and the methods have achieved explanation results that are more visually pleasing compared with previous methods. However, the results are probably limited when the segmentation map's quality is poor. Petisiuk et al. proposed the RISE method (Petsiuk et al., 2018) , which simulates the feature's absence using randomized masks and measures its response to each masked instance. Because of its random masking strategy, RISE requires a considerable number of feedforward executions and suffers from local noise. The meta learning approach that tries to maximize the interpretability of a learning model is used in some studies (Ribeiro et al., 2016; Fong & Vedaldi, 2017) . One study (Ribeiro et al., 2016) employed superpixel-wise random samples around the instance and an approximate linear decision model. Fong & Vedaldi (2017) proposed an optimized framework that learns a minimum perturbation mask from the corresponding response to its output neuron. However, such frameworks often fail to optimize their result because of its sensitivity to various types of models and instances. Unlike the white-box approaches, the black-box methods are inherently model agnostic; i.e., they are applicable to any learning model, because they rely only on the output values, regardless of the internal workings of the classification models.

The fine-grained classification is to recognize subordinate classes of a base class such as species of birds and different models of cars and planes. Most of recent works use the deep CNN models and propose better loss function. Shi et al. (2018) proposed a generalized large-margin (GLM) loss to reduce between-class similarity and within-class variance. The contrastive loss (Sun et al., 2014) and triplet loss (Schroff et al., 2015) have also been proposed. Qiu et al. 2018 (Seo et al., 2018) , fish (Qiu et al., 2018) , vehicle (Li et al., 2019) , plant (Lin et al., 2019) , leukemia (Sipes & Li, 2018) , and plankton (Lee et al., 2016) .

3. PROPOSED APPROACH:

GLAS method. Given an image I, we define a set of search points Ω = (µ 1 , µ 2 , ..., µ k×k ) by the centers of k × k grids overlaid upon I, as shown in Figure 2 . For a given class label y and a specific search point µ i , the prediction score f l (µ i ) by Gaussian light can be written as

EQUATION

Figure 2 : Overview of the GLAS method.

Where denotes element-wise multiplication, and G(µ i , σ l ) is a Gaussian distribution with mean µ i and standard deviation σ l . Eq 1 simulates light projected on a specific part of the image to measure the contribution of the local pattern. It is also possible to define the score based on the inverse-Gaussian mask; i.e., the shadow is given by the following equation:

EQUATION

where

G (µ i , σ l ) = 1 -G(µ i , σ l ).

Here,λ is a constant, and we empirically set it to 10 5 to avoid f s (µ i ) being 0. We use a weighted fusion to define the saliency score S(x j ) for a pixel x j as

EQUATION

where exp(-

D(xj ,µi) σ 2 spatial

) is a spatial weighting factor; here, D(a, b) denotes the distance between a and b. Eq3 represents the Gaussian mixture-based weighted fusion. The high flexibility of the visual explanation can be achieved by adjusting the scale parameter for each Gaussian mask.

Recursive GLAS (RGLAS) method. GLAS tends to high-light the most discriminative clue. To discover the various evidences that lead to the classifier's decision, we propose a simple schema called RGLAS. Figure 3 shows the key idea of RGLAS: to prevent revisits to the search points related to the most discriminative fea-tures that have already been found, leading to extraction of the next important features. This mechanism also helps in discovering multiple instances in an for each pixel x j do 4:

S(x j ) = 1 |Ω| µi∈Ω exp(- D(xj ,µi) σ 2 spatial )f l (µ i )f s (µ i ) 5:

end for

6: B = S > t 2 7:

for each serach point µ i do 8:

If(B(µ i )==1) remove µ i from Ω 9:

I = I (1 -S) 10:

end for 11:

O+ = normalizeS image. The RGLAS algorithm starts by constructing the saliency map S (lines 3-4). We compute the binary map of S using the threshold value t 1 = 0.8 (line 5) and eliminate search points located in the positive region of the binary map (lines 6-7). The input image is updated using the previous input and the inverse saliency map (line 8). We define a simple stop condition, as formu-lated in line 9, with t 2 = 5. We found that as the iteration increases, mean(S) tends to increase but f (y|I) decreases, guaranteeing that the stop condition occurs consistently.

4. EXPERIMENTAL RESULTS:

The experiments were conducted on an Intel Core i7-7800X with a 3.50 CPU, 32 GB of memory, and a GTX 1080 Ti GPU. We aimed to evaluate quantitatively and qualitatively the salient explanation capabilities of GLAS and existing explanation models. Salient explanation for fine-grained classification tasks.

The GLAS provides us with fine-level visual clue identification, enabling the salient explanation.

To demonstrate the effectiveness of scalability, we employed CUB200 (Wah et al., 2011) set of airplanes. We used the basic procedure of transfer learning using ResNet-50 pretrained on ILSVRC. The resulting networks of CUB200, Stanford Cars and Aircraft yielded top-1 accuracies of 76.11%, 92.01%, and 80.59%, respectively. GLAS can pinpoint more detailed clues by adjusting its scale parameters.Figure 4 shows the visual comparison. In the experiment, we used an equal sigma value of 3.0 for σ l , σ s , and σ spatial . Unlike other methods, we can see that GLAS consistently pinpoints meaningful patterns of birds. For example, in the case of the Northern flicker, an instance has characteristics such as the red spot below the eyes and black dots on the body. GLAS surprisingly pinpoints two characteristics of the Northern flicker with the scale 3.0; however, the other methods only discover the location of the instance and fail to explain the detailed patterns.

In this regard, Grad-CAM and RISE tend to explain only global significance, such as the target's location. Numerous visual examples are available in the Supplementary Materials.

We conducted another experiment measuring how much the pinpointed clues affect the class decision. In Figure 5 , original images are perturbed by the inverse heat map, and the amount that the score drops is recorded. As expected, the class score dropped rapidly, ensuring the significance of the pinpointed features. Figure 6 shows the European goldfinch characterized by red face and yellow spot on the wings used for demonstrating coarse-to-fine controls. GLAS adjusts the standard deviation. Deconv adjusts the occlusion mask size. The RISE adjusts the size of the initial mask.

The most prominent observation is that the heat maps from RISE and Deconv are very noisy and less accurate in identifying the most discriminative parts of the relevant object. The failure case analysis in Figure 7 deserves an attention. The second row unveils an interesting behavior of CNN through failure cases. The first and second images belonging to the Red-legged Kittiwake was incorrectly classified into Pigeon guillemot with 34.09% and 11.59% probabilities, respectively. The salient explanation capability of GLAS allows us to understand that the CNN misclassified the images into the Pigeon guillemot by looking at the red leg. Figure 8 shows the visual explanations on Aircraft and Stanford Cars benchmarks, respectively. For Aircraft examples, GLAS consistently pinpointed propellers of wings for the class "Yu 12". For Stanford Cars dataset, we found that CNN changed its gaze consistently according to poses of Car. When the front is shown for the car class 138, the grille part is highly probable to be pinpointed. When the back side is shown, lamps and wheels are pinpointed. Note that these behaviors of CNNs can be unveiled only when the salient explanation is available. Evaluation on target localization.

We performed quantitative evaluations on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) (Deng et al., 2009) . As a metric, we employed the pointing game (PT) presented in the study (Ribeiro et al., 2016) . The PT purely measures the spatial selectiveness of the continuous visual saliency map. In the evaluation, the PT detects the maximum intensity point on the saliency map, and a Hit is recorded if the maximum point is in the ground-truth annotation; otherwise, a Miss is recorded. The accuracy is calculated using Acc = Hit Hit+M iss . Because multiple maximum points often arise, we em-ployed the threshold value T > 0.95 to generate binary blobs, and then we used the centroid of the biggest blob as the localization point. We empirically set the scale parameters σ l =5 and σ s =3, with σ spatial =6. Table 1 shows the execution time of GLAS according to the number of grid search points. The result tends to show favorable PT scores as the number of the search points grows. It starts to become saturated after k = 15 in terms of performance, even as the execution time continues to grow. Table 2 illustrates the results of comparisons with the existing methods on the ILSVRC validation dataset. GLAS achieves the best result in terms of PT score. GLAS is 13 times faster than RISE even with the higher PT score. This is because a considerable number of perturbed images using a randomized masking process are necessary for reliable visual explanation in RISE. When RISE is forced to use 450 (225 × 2) perturbed images, identical to the number used by GLAS, we observed that the PT score of RISE drops from 0.907 to 0.869. This observation tells us that GLAS perturbs and localizes the important features efficiently. We separately evaluated the cases in which the object is small in the PT-small column of 2. Like a related study (Cao et al., 2015) , we consider an object to be small if the total area of the bounding box of the given class is smaller than one quarter of the size of the image. Even though all models encountered a performance drop, GLAS still beats other models.

In our work, GLAS operations can be used together or independently. The results in the last three rows of Table 2 show an ablation study on GLAS by measuring model performance with either Gaussian lighting or shadowing suppressed. Table 2 shows the performance of the ablated GLAS is comparable to that the state-of-the-art methods, outperforming all methods except Grad-CAM, RISE, and fused GLAS. Figure 9 provides a visual comparison of the methods. GLAS and Grad-CAM clearly highlight the important region related to a given class, whereas MASK and RISE suffer from nontrivial local noise. The advantage of GLAS is obvious without the noise and produces a visualization map that is highly interpretable. Because the GLAS map consists of Gaussian mixture clues, it identifies the most important area without being distracted by meaningless clues.

Multiple evidence discovery by the recursive process. Since the Automated Anatomical Labeling (AAL) [5] map exists, all MRIs are spatially normalized based on AAL, and then we considered the centroid of AAL as the search points. In this work, 3D-GLAS method were considered to perturb 3D-MRI data, and we empirically set σ=10 and σ=15. The validation MRIs of AD category are fed to 3D-GLAS, then the entire saliency maps of validation MRIs were linearly integrated and normalized. For this reason, the highlighted biomarkers have statistical significance. Since the centroids of each AAL segment are considered as search points, we mark out impact of each biomarker to its corresponding AAL segment directly. In Figure 25 , the hippocampus, amygdala and temporal inf were selected as the important biomarkers for the accurate classification of AD. These biomarkers have previously been known to be closely related with dementia in many existing studies [1][2][3][4] . In particular, the hippocampus, a brain region for learning and memory, is one of the first brain biomarker, which is affected by AD and undergoes severe structural changes as the disease progresses [4] Figure 15 : Crested auklet. The uniform characteristic is orange beak. 

:

CONCLUSIONIn this study, we proposed a visual explanation method called GLAS for the black-box model. Our method is in-spired by the natural light and shadow phenomena and provides a simple yet robust way to perturb an input in-stance. The GLAS presented the ability of fine-level visual explanation at various scales through the adjustment of the Gaussian scale. Additionally, we showed the broad applicability of GLAS to various tasks. In experiments, GLAS showed state-of-the-art performance and efficient computing time. For a future work, we plan to improve the heat map by optimizing the scale parameters of the Gauss-ian mask adaptively on an image instance. A deeper theo-retical analysis of GLAS is also needed.

Figures and Tables:

Figure 3: Framework of RGLAS. The GLAS instances are repeated until all discriminative patterns related to a given class have been dis-covered.

Figure 4: Visual comparison with the existing models. From top to bottom: red-winged blackbird, Northern flicker, European goldfinch, and crested auklet. We introduce the uniform characteristics of each bird: first row (red wings), second row (red below the eyes and black spots on the body), third row (red face and yellow on the wings), and fourth row (orange beak).

Figure 5: From left to right: original image, heat map, and image perturbed by the inverse heat map along with the class probabilities. Green-tailed towhee and red-winged blackbird.

Figure 6: Visual comparisons of GLAS, RISE, and Deconv according to their parameters controlling the locality. The example is the Eu-ropean goldfinch characterized by red face and yellow spot on the wings. GLAS adjusts the standard deviation. Deconv adjusts the occlusion mask size. The RISE adjusts the size of the initial mask.

Figure 7: Unveiling the feature selection behaviors of CNN. The top two rows illustrate two bird classes, Red-legged Kittiwake and Pi-geon guillemot. Despite its name, the Red-legged Kittiwake were consistently highlighted on faces while the Pigeon guil-lemot was highlighted on the red legs Figure 8: Visual explanations on Aircraft and Stanford Car benchmarks.

Figure 9: Visual comparison of the classdiscriminative capability of MASK, Grad-CAM, RISE, and GLAS by varying the object classes.

Figure10: Recursive process. This algorithm discovers the relevant parts of a given class in order of significance. We used the adaptive stop condition mentioned in Section 3.

Figure 11: Visual explanation examples produced by GLAS for the image captioning model.

Figure 16: Eared grebe. The uniform characteristic is red eyes.

Figure 17: Red headed woodpecker. The uniform characteristic is red head.

Figure 18: Red bellied woodpecker. The uniform characteristic is black and white barred patterns on their back, wings and tail. Adult males have a red cap going from the bill to the nape; females have a red patch on the nape and another above the bill.

Figure 19: Pileated woodpecker. The uniform characteristic is red crest.

Figure 20: Prairie warbler. The uniform characteristic is dark streaks on the flanks and head.

Figure 21: Myrtle warbler. The uniform characteristics are yellow crown, rump and flank patches.

Figure 22: European goldfinch. The uniform characteristics are red face and broad yellow bar on the wings.

Figure 23: Gold Winged Warbler. The uniform characteristics are yellow crown and wing patch.

Figure 24: Rose breasted grosbeak. The uniform characteristic is bright red-red patch on the breast.

proposed a method based the sqeeze-and-excitation atention model. Peng et al. 2018 used both the object-level attention and part-level attention. The literatures treated various types of objects. Shi et al. (2018) used birds, cars, and airplanes datasets. Other objects include fashion

Table content: <table/>

PT scores according to the number of search points Search points k × k 12 × 12 15 × 15 22 × 22 25 × 25 30 × 30

Table content: <table><tr><td>PT score</td><td>0.905</td><td>0.912</td><td>0.914</td><td>0.913</td><td>0.913</td></tr><tr><td>Time(s)</td><td>0.398</td><td>0.652</td><td>1.435</td><td>1.909</td><td>3.95</td></tr></table>

Quantitative comparisons with existing models using the PT on the ILSVRC validation

Table content: <table><tr><td>data using ResNet50</td><td/><td/><td/></tr><tr><td>Methodn</td><td>PT</td><td colspan="2">PT-small Time(s)</td></tr><tr><td>Grads (Simonyan et al. 2013)</td><td>0.773</td><td>0.604</td><td>0.128</td></tr><tr><td>Deconv (Zeiler et al. 2014)</td><td>0.750</td><td>0.584</td><td>0.117</td></tr><tr><td colspan="2">Grad-CAM (Selvaraju et al. 2017) 0.901</td><td>0.754</td><td>0.183</td></tr><tr><td>Deconv (Zeiler et al. 2014)</td><td>0.809</td><td>0.688</td><td>2.64</td></tr><tr><td>LIME (Ribeiro et al. 2016)</td><td>0.766</td><td>0.645</td><td>15.11</td></tr><tr><td>RISE (Petisiuk et al. 2018)</td><td>0.907</td><td>0.787</td><td>8.11</td></tr><tr><td>MASK (Fong et al. 2017)</td><td>0.841</td><td>0.711</td><td>16.38</td></tr><tr><td>GLAS (light)</td><td>0.889</td><td>0.782</td><td>0.328</td></tr><tr><td>GLAS (shadow)</td><td>0.877</td><td>0.751</td><td>0.328</td></tr><tr><td>GLAS (fusion)</td><td>0.912</td><td>0.792</td><td>0.612</td></tr></table>

illustrates the mean intersection over union (IOU) scores of the proposed recursive process. Because the visual saliency map consists of continuous intensity values, the mean IOU scores were measured with varying thresholds, from 0 to 1.0. In particular, significant progress occurs at the second iteration. Higher IOU scores over the threshold values indicate that the object regions are

Table content: <table/>

IOU scores of the proposed method on the ILSVRC validation data as the iteration increases. "A" represents the final iteration under the adaptive stop condition In the fusion results shown in the last column of Figure10, we can see that multiple instances and multiple evidences are well discovered. Visualization of the image captioning model.Image captioning is a challenging task for which both computer vision and natural language processing techniques should be considered. We constructed the image captioning model based on publicly available implementations for which the fine-tuned InceptionV3-based image and long short-term memory-based language models are considered. Figure11shows some visual explanation results from the image captioning model to demonstrate the applicability of GLAS. GLAS shows the capability to localize visual concepts such as objects (man, bicycle, ball, boy, hurdle, and dog) and actions (riding, playing, and jumping).We consider a neuroimaging classification problem to show applicability of GLAS in a medical imaging domain. We employed 3D-magnetic resonance imaging (MRI) scans reflecting 199 Alzheimer's Disease (AD) vs. 230 healthy Normal Control (NC) from the Alzheimer's Disease Neuroimaging Initiative (ADNI), which is publicly available. A 3D-CNN was employed for classification of AD vs. NC. Due to limited dataset size, 3D-MRIs are spatially normalized based on template brain image and unsupervised learning technique (Convolutional auto-encoder) is applied before supervised learning. The overall architecture is comprised of three 333 Conv layers with 10 filters each, two FC layers with 32 and 16 nodes each, and softmax activation; each of Conv layers is followed by ReLU and 222 max-pooling; Gaussian dropout with a dropping ratio 0.8 is applied in between Conv layers; in the FC layers, we used SELU activation for speeding up learning and taking normalization effect in the FC layers. 5-fold cross validations were conducted to evaluate the classification model. The mean accuracy was 85.31%.

Table content: <table><tr><td>Iteration</td><td/><td/><td>Threshold</td><td/></tr><tr><td/><td>0.1</td><td>0.3</td><td>0.5</td><td>0.7</td><td>0.9</td></tr><tr><td>1</td><td colspan="5">0.53 0.51 0.40 0.31 0.09</td></tr><tr><td>2</td><td colspan="5">0.54 0.56 0.51 0.39 0.07</td></tr><tr><td>3</td><td colspan="5">0.55 0.56 0.52 0.37 0.06</td></tr><tr><td>A</td><td colspan="5">0.55 0.59 0.55 0.38 0.06</td></tr><tr><td>uniformly highlighted.</td><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
The paper introduces a new method in the family of local perturbation-based interpretations for deep networks and more specifically for fine-grained classification tasks. Compared to other saliency map methods (gradient-based, propagation-based, etc), this family has the advantage of needing only black-box access to the model. The introduced method GLAS, scans over an image and lights/shadows each part of the image to assign an importance score to different regions of the image based on the change in model*s prediction. The motivation of this work is to increase the inherently low speed of (some) methods in this family and to give better explanations by I vote for rejecting this paper as the contributions to what already exists in the literature are not clear and the provided experimental results are not convincing. Compared to previous perturbation-based methods, the main advantage seems to be speed. There are perturbation-based methods that do not suffer from low speed e.g. Dabkowski & Gal and give real-time perturbation-based saliency maps. The authors do not mention this work (and similar works) and do not compare both their speed and their performance against it. One important problem (probably the most important) with the perturbation-based saliency maps is the fact that the perturbations might push a given image out of the true data manifold and therefore give an invalid interpretation of the model. The authors do not discuss the matter and how their method would address this issue. Intuitively, the introduced algorithm, more specifically the RGLAS algorithm, seems to suffer from this issue not any less than other existing methods. The experimental results seek to demonstrate the superiority of the introduced method over rival methods. The justification behind the provided visual examples is their focus on more discriminative features (in human eyes). This does not necessarily mean that a given saliency map is more *truceful*; i.e. having a more visually appealing saliency map has nothing to do with a more truthful explanation of a model*s decision making. The results focused on the mistakes of the model seem more convincing and interesting. The objective results first focus on the target localization metric which has traditionally been used in the literature. Although it is much faster to execute, the introduced method is only marginally superior to other methods. The most important problem, however, is that as mentioned above, there are fast methods in the literature and therefore a fair objective comparison is should include other methods as well. Secondly, the IOU measure is used. GLAS is not compared to other methods in this metric. The authors mention the effectiveness of their work for *fine-grained* classification tasks while throughout the paper there is no convincing evidence or discussion that the method is curated for such tasks. As mentioned above, changing the scale parameter for getting more visuall appealing saliency maps is not enough evidence. All in all, the true contribution of this work to other existing methods in this family is not enough for this venue. A few questions and suggestions: * How should one adjust the scale parameter? In other words, what is the hyper-parameter search scheme for this method which would make it robust against the human-biased choice of hyper-parameters which would result in visually more appealing saliency maps but not necessarily explain the model? * Explanation of RGLAS is not clear. * For a general reader, metrics such as IOU should be explained more clearly. * The paper has many many typing errors.