Salient Explanation for Fine-grained Classification
Authors: Kanghan Oh
Sungchan Kim
Il-Seok Oh
Venue:  (2020)
Keywords: Visual explanation
XAI
Constitutional Neural Network

Abstract: Explaining the prediction of deep models has gained increasing attention to increase its applicability, even spreading it to life-affecting decisions. However there has been no attempt to pinpoint only the most discriminative features contributing specifically to separating different classes in a fine-grained classification task. This paper introduces a novel notion of salient explanation and proposes a simple yet effective salient explanation method called Gaussian light and shadow (GLAS), which estimates the spatial impact of deep models by the feature perturbation inspired by light and shadow in nature. GLAS provides a useful coarse-to-fine control benefiting from scalability of Gaussian mask. We also devised the ability to identify multiple instances through recursive GLAS. We prove the effectiveness of GLAS for fine-grained classification using the fine-grained classification dataset. To show the general applicability, we also illustrate that GLAS has state-of-the-art performance at high speed (about 0.5 sec per 224$\times$224 image) via the ImageNet Large Scale Visual Recognition Challenge. 

Under review as a conference paper at ICLR 2020
SALIENT EXPLANATION FOR FINE-GRAINED CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Explaining the prediction of deep models has gained increasing attention to in- crease its applicabilityy even spreading it to life-affecting decisions. However there has been no attempt to pinpoint only the most discriminative features con- tributing specifically to separating different classes in a fine-grained classification task. This paper introduces novel notion of salient explanation and proposes a simple yet effective salient explanation method called Gaussian light and shadow (GLAS), which estimates the spatial impact of deep models by the feature pertur- bation inspired by light and shadow in nature. GLAS provides a useful coarse- to-fine control benefiting from scalability of Gaussian mask. We also devised the ability to identify multiple instances through recursive GLAS. We prove the effectiveness of GLAS for fine-grained classification using the fine-grained clas- sification dataset. To show the general applicability, we also illustrate that GLAS has state-of-ther performance at high speed (about 0.5 see per 224x224 image) via the ImageNet Large Scale Visual Recognition Challenge.
INTRODUCTION
Over the last several years, convolutional neural networks (CNNs) LuCunn et al. 2015) have achieved superior performance in various computer vision tasks, including image classification (He et al.] 2216] Shi et al. 2018), object detection (Oh et al., 2017 Zhou et al. 2016). semantic segmen- tation (Pathak et al. 2015). and image captioning (Xu al. 2015). Despite these dramatic advances, the opacity of CNNs makes it difficult to understand why they reach particular decisions, limiting the ability widen their application to various fields.
In general, the visual interpretation of deep learning models is understood as estimating the impact of a particular neuron activation related given input instance. In white-box approach, architectural modification of the classification model (Bach et al. 22055; Dong et al. 2017 Mahendran & Vedaldi
2016 Selvaraju et al., 2017; Simonyan et al. 2013 Springenberg et al. 2014] Zhou et al. 2016
Zeiler & Fergus/ 2013) access to specific layers (Bach et al.) 2015 Selvaraju et al.) 2017 Zhang
ct al. 2016) is inevitable (Petsiuk et al.] 2018), resulting in severe limitation of application. In
contrast, the black-box approach (Seo al. 2018 Petsiuk et al. 2018 Ribeiro al. 2016 Tian &
Cail 2017 Zeiler & Fergus 2013) Zintgraf et 201]
& Vedaldi (017)) aims to be inherently model agnostic. Its main concerns are how to perturb an input image and draw the model's response on the perturbed instance to the final heat map. For example, the Randomized Input Sampling for Explanation (RISE) method (Petsiuk et al. 2018) perturbed an image with a randomised mask to measure the importance of pixels and then linearly fused all importance from several thousand masks.
The conventional black-bon -bx methods employed unnatural and fragile perturbation schemes such as single colour out (Seo et al. 20188, Petsiuk et al. 2018 Ribeiro et al. 2016 Zeiler & Fergus 2013). random noise (Tian & Cail |017] Zintgraf et al. 2017; Fong & Vedaldi 2017) and smoothing (Fong
& Vedaldi 2017). These perturbation schemes have several limitations. First, they are deficient in pinpointing only the most discriminative, i.e., salient features that are essential for the fine-grained classification tasks where the between-clase shape similarity is very high; for example, pinpointing only the red face of Red-faced Cormorant in the bird classification task in Figure i crucial for explaining why a deep learning model classifies the image as Red-faced Cormorant. Second, the conventional perturbation schemes highly suffer from local noise and, thus, fuse maps from

---

Under review as a conference paper at ICLR 2020
Red faced Cormorant
Sigma-6
Sigma=5
Sigma=4
Sigma=3
Figure I: Salient explanation. As the value of decreases, the heat map concentrates more and more on the most discriminative part of the relevant objects, the bird's face in this case. The salient expla- nation is very important for variety of tasks such the fine-grained classification and biomarker discovery in medical image. Note that the salient explanation is possible due to our idea of adopting Gaussian mask.
siderable number of perturbations for a reliable explanation. This the main cause slowness with conventional black-box methods.
Inspired by the lighting and shadowing phenomena in nature, we propose a simple yet effective black-box method, called Gaussian light and shadow (GLAS), which simulates feature perturbation as the presence or the absence of light at the pixel level of an image. The primary idea of GLAS is to perturb an input image by the Gaussian mask (light) and Hnverse-Gaussinn mask (shadow) and, then, record the responses of the perturbed images. GLAS uses simple grid search; once completed over the entire image, the response maps are fused construct the final heat map. The fusion mimics the Gaussian mixture. The proposed method has several advantages compared with other blck-box methods (Petsiuk al., 2018 Zintgraf et al., (017)) [ong Fong Vedaldi (017). First, GLAS provides scalability of explanation that we can achieve by adjusting the variance parameter of the Gaussian mask. The scalability makes it possible to pinpoint clues for the salient explanation, which is not feasible with the nonparameterized approaches (Mahendran & Vedaldi 2016] /hou et al.) 2016 Zeiler & Fergus, 2013 Petsiuk et al., 2018 Zintgraf et al. [017]] Fong & Vedaldi (017). The salient explanation is valid for explaining fine-graing grained classifications, such as classifying bird species in a CUB200 dataset, involving large between-class similarity and significant within-class variance. Figure I shows image of the Red-faced cormorant species that identifying the face color. It illustrates that GLAS adjusts its gaze from the body to the red face as the scale parameter decreases and finally pinpoints the red multiplication operation with the Gaussian mask at a specific search point simulates the gradual dimming effect as going farther from the center. We argue that because of this characteristic, significantly reduced number of perturbations is sufficient. GLAS can process an image much faster than conventional methods.
discriminate by
around the eye. Second, our pixel-wise
To summarize, the contributions of this paper are as follows: We introduce a novel notion of salient explanation which is critical in explaining the fine-grat fine-grained classification tasks. We propose simple yet efficient black-box method, GLAS, which provides an easy way to perturb an input image based on Gaussian lighting and shadowing. (1) GLAS is fast because of the smoothly varying shape of the Gaussian mask, which generates visual explanation up to order of magnitude faster than other black-box methods. (3) We show the broad applicability of GLAS to various other tasks: object localization and visual captioning. Quantitative comparisons show that GRAS is superior to conventional methods.
2 RELATED WORKS
The white-box approach heavily uses the network's internal information, such as gradients or feature maps of specific layers. A gradient can indicate how much small change in pixel influences the class output (Springenberg et al., 2014). For cxample,[imonyan
Simonyan et al. (2013) proposed the gradient- based model, which directly mapped saliency values the original space. Additionally, Zeiler & Fergus (0133) proposed a deconvolution method. the method, the forward signal is reversed at a neuron and backpropagated to the input space. The study (Bach al.] 2015) proposed the layer- layer-wise relevance propagation method, in which the prediction in the output layer is decomposed into pixel- wise relevance values and backpropagated until it satisfies the conservation rule. (aame et al. (2016)
2

---

Under review as a conference paper at ICLR 2020
emphasized the importance of quantitative evaluation and provided a rigorous comparison of the pre- viously mentioned methods; these approaches are extensively reviewed in Samek et al. (2016). The visual feature maps provide important clues for the explanation, which some techniques exploit. The technique (Zhou et al., 2016) called class activation mapping (CAM) is accomplished by weighted fusion of visual feature maps and requires the modification of CNN architecture, replacing the fully connected layer with the global average pooling. Grad-CAM (Selvaraju et al. 2017). an extended version of CAM, is applicable to a broader range of CNNs. The previously mentioned techniques modify the model's 's internal operations or rely on the model's s internal values; thus, they are model dependent.
The black-box approach measures the response change of the base model when the input instance is spatially perturbed, and this change can be regarded as the significance of the classifier's 's decision. The study (Robnik Kononenko 2008) simulates feature perturbation based on marginal probabil- ity, and several studies have extended and improved this method (Tian & Cai 2017 Zintgraf et al.) 2017]. For the CNN-based architectures, the study (Zintgraf et al. 2017) proposed the conditional sampling approach. The method considers that given pixel value highly depends pixels and that multivariate analysis excludes a rectangular region rather than a single pixel. Because
neighboring
pixel-wise perturbation method such as random noise (Zintgraf et al. 2017 Fong & Vedaldi 2017) was considered, pixels are highly vulnerable to adversarial attack.
Several techniques (Seo et al. 2018: Ribeiro et al. 2016 Tian & Cail 2017) aim at region-wise per- turbation approaches, rather than using pixels. A study (Tian & Cail 2017) improved the conditional sampling method using the superpixel algorithm, making it more robust from local noise than Zint- graf's 's method. Additionally, the superpixel segmentation technique (Seo et al., 2018; Ribeiro et al., 2016 Tian & Cail 2017). In these methods, high-level segments, rather than pixels (Zintgraf et al. 2017) oversegmented regions (Tian& & Cail 2017), are used to perturb the feature of instance, and the methods have achieved explanation results that are more visually pleasing compared with previous methods. However, the results are probably limited when the segmentation map's quality is poor. Petisiuk et al. proposed the RISE method (Petsiuk et al.) 2018). which simulates the feature's absence using randomized masks and measures its response to each masked instance. Because of its random masking strategy, RISE requires a considerable num- ber of feedforward executions and suffers from local noise. The meta learning approach that tries to maximize the interpretability of learning model is used in some studies (Ribeiro et al.) 2016]] Fong & Vedaldi 2017). One study (Ribeiro et al. 2016) employed superpixel- -wise random samples around the instance and an approximate linear decision model. Fong & Vedaldi (217)) proposed an optimized framework that learns a minimum perturbation mask from the corresponding response to its output neuron. However, such frameworks often fail to optimize their result because of its sen- sitivity to various types of models and instances. Unlike the white-box approaches, the black-box methods are inherently model agnostie; i.e., they are applicable any learning model, because they rely only on the output values, regardless of the internal workings of the classification models.
used in existing methods
The fine-grained classification is to recognize subordinate classes of a base class such as species of birds and different models of cars and planes. Most of recent works use the deep CNN models and propose better loss function. Shi et al. (2018) proposed a generalized large-margin (GLM) loss to reduce between-class similarity and within-class variance. The contrastive loss (Sun et al.) 2014) and triplet loss (Schroff et al., 2015) have also been proposed. Qiu et al. 2018 proposed a method based the wanpee---nndexeittattie and part-level attention. The literatures treated various types of objects. [Shi et al. (2018) used birds, cars, and airplanes datasets. Other objects include fashion (Seo et al., 2018), fish QQuu et al. 2018)
atention model. Peng et al. 2018 used both the object-level attention
vehicle (Li et al. 2019), plant (Lin et al. 2019), leukemia (Sipes & Li 2018). and plankton (Lee jet al. 2016).
PROPOSED APPROACH
GLAS method. Given an image I, we define a set of search points Ω = (μr, ... xk by the centers of k l grids overlaid upon I, as shown in Figure For a given class label y and a specific search point Ps the prediction score f((m)) by Gaussian light can be written as
f1(44) = P(y] o G())
3

---

Under review as a conference paper at ICLR 2020
()) 1OG ())
Responses target ("Tiger cat")
Learned model
fi
f
(Black-box]
0.54
0.56
0.67
Image
INPUT
0.02
0.67
Initial search points
Spatial weight 6, g. Spatial
Guassian light
Guassian
Weighted fusion
shadow
spatial
Outcome
Figure 2: Overview of the GLAS method.
Where ( denotes element-wise multiplication, and G(Us; σ1) is Gaussian distribution with mean M4 and standard deviation dj. Eq [ simulates light projected on specific part of the image to measure the contribution the local pattern. It is also possible to define the score based on the inverse-Gaussia mask; i.e., the shadow is given by the following equation:
fa(%) IP(w P(y o G" ot)) +
(2)
where G' G"(,, 01) = G(4, 01). Here,A is constant, and empirically set it to 10 to avoid (pss) being O. We use a weighted fusion to define the saliency score S(x;) for pixel æj as
D(a ({(p)))s(p)) Sspatiat
S(z,) = S22
Σ BER
exp(
where expl D(2s.m))) is spatial weighting factor, here, ((a, b) denotes the distance between 2 spatial
and b. Ed represeets the Gaussian mixture-hased weighted fusion. The high flexibility of the visual explanation can be achieved by adjusting the scale parameter for each Gaussian mask.
Recursive GLAS (RGLAS) method. GLAS tends high-light the most discriminative clue. To discover the various evidences that lead to the classifier's s decision, we propose a simple schema called RGLAS. Figure 3 shows the key idea of RGLAS: prevent revisits the search points related to the most discriminative fea-tures that have already been found, leading to extraction of the next important features. This mechanism also helps in discovering multiple instances in an
iteration
Itorabinn
INPUT
GLAS
iteratton
Iteration
Recursive process
Restricted search points
Inverse Saliency
mapping
Figure 3: Framework of RGLAS. The GLAS instances are repeated until all discriminative patterns related to given class have been dis-covered.
4

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Recursive GLAS
Require: image 1, target class e Ensure: Final saliency map O
1: 0=0; (Mz, M2, ... k)
2: while (/menn(S)/[en
<t2
3: for each pixel 3. do
4:
S(z,) [ Eu,e0 exp( end for
D(m) (f(()))f((1)))
spatial
5:
6:
B= l2
7:
for each serach point ja, do [((((u))=l) remove μr from Ω 1=10 () end for
8:
9:
10:
11:
0+ = normalizes
image. The RGLAS algorithm starts by constructing the saliency map S (lines 3-4). We compute the binary map of S using the threshold value f 0.8 (line 5) and eliminate search points located in the positive region of the binary map (lines 6-7). The input image updated using the previous input and the inverse saliency map (line 8). We define simple stop condition, as formu-lated in line 9, with (2 = 5. We found that as the iteration increases, mean(S) tends to increase but f'vyl (T) decreases, guaranteeing that the stop condition occurs consistently.
4 EXPERIMENTAL RESULTS
The experiments were conducted on an Intel Core 17-7800X with a 3.50 CPU, 32 GB of memory, and GTX 1080 Ti GPU. We aimed to evaluate quantitatively and qualitatively the salient explanation capabilities of GLAS and existing explanation models.
Salient explanation for fine-grained classification tasks.
The GLAS provides us with fine-level visual clue identification, enabling the salient explanation.
To demonstrate the effectiveness of scalability, employed CUB200 (Wah et al. 2011), Stanford Cars (Kaause et al. 2013), and Air-craft (Maji et al. 2013) benchmarks that have been used for the tine-grained image classification tasks. The CUB200 dataset consists of 11,788 images of 200 bird species. The Stanford Cars dataset includes 8, 1,14 training and 8,041 test images with 196
classes. The Aircraft dataset is set of 10,000 images with 100 classes reflecting a fine-graine
Input
Grad-CAM
RISE
GLAS
Input
Grad-CAM
RISE
GLAS
Red vingeed Stachtrd
Nettenn Mecker
Inmpeen golcfinch
Cresled miklet
Figure 4: Visual comparison with the existing models. From top bottom: red- red-winged blackbird, Northern flicker, European goldfinch, and crested auklet. We introduce the uniform characteristics of each bird: first row (red wings), second row (red below the eyes and black spots the body), third row (red face and yellow on the wings), and fourth row (orange beak).
5

---

Under review as a conference paper at ICLR 2020
set of airplanes. We used the basic procedure of transfer learning using ResNet-50 pretrained on ILSVRC. The resulting networks of CUB200, Stanford Cars and Aireraft yielded top-1 accuracies of 76.11%, 92.01%, and 80.59%, respectively. GLAS can pinpoint more detailed clues by adjusting its scale parameters. Figure 4 shows the visual comparison. In the experiment, we used an equal sigma value of 3.0 for di, a. and spatial- Unlike other methods, tently pinpoints meaningful patterns of birds. For example, in the case of the Northern flicker, an instance has characteristies such the red spot below the eyes and black dots on the body. GLAS surprisingly pinpoints two characteristics of the Northern flicker with the scale 3.0; however, the other methods only discover the location of the instance and fail to explain the detailed patterns. In this regard, Grad-CAM and RISE tend to explain only global significance, such the target's location. Numerous visual examples are available the Supplementary Materials.
see that GLAS consis-
We conducted another experiment measuring how much the pinpointed clues affect the class deci- sion. In Figure 5 original images are perturbed by the inverse heat map, and the amount that the score drops recorded. As expected, the class score dropped rapidly, ensuring the pinpointed features. Figure [6] shows the European goldfinch characterized by red face and yel- low spot on the wings used for demonstrating coarse-to- fine controls. GLAS adjusts the standard deviation. Deconv adjusts the occlusion mask size. The RISE adjusts the size of the initial mask. The most prominent observation is that the heat maps from RISE and Deconv are very noisy and less accurate in identifying the most discriminative parts of the relevant object. The failure in Figure F deserves an attention. The second row unveils an interesting behavior of CNN through failure cases. The first and second images belonging the Red-legged Kittiwake was incorrectly classified into Pigeon guillemot with 34.09% and 11.59% probabilities, respectively. The salient explanation capability of GLAS allows us understand that the CNN misclassified the images into the Pigeon guillemot by looking at the red leg. Figure 8 shows the visual explanations on Aircraft and Stanford Cars benchmarks, respectively. For Aircraft examples, GLAS consistently pinpointed propellers of wings for the class Yu 12". For Stanford Cars dataset, we found that CNN changed its gaze consistently according to poses of Car. When the front is shown for the car class 138, the grille part is highly probable to be pinpointed. When the back side is shown, lamps and wheels are pinpointed. Note that these behaviors of CNNs can be unveiled only when the salient explanation is available.
significance of
analysis
GLAS
Sig=4
Sig
5ig 3
European goldfinch
0.87
0,21
RISE
Reduced =10
Reduced - 20
Reduced
Deconv
0.97
0.17
Rect=
Recto5
Figure 6: Visual comparisons of GLAS, RISE, and Deconv according to their parameters controlling the locality. The example is the Eu-ropean goldfinch characterized by red face and yellow spot on the wings. GLAS adjusts the stan- dard deviation. Decony adjusts the occlusion mask size. The RISE adjusts the size of the initial mask.
Figure From left to right: original image, heat map, and image perturbed by the inverse heat map along with the class probabilities. Green-tailed towhee and ren-winged blackbird.

---

Under review as a conference paper at ICLR 2020
Success cases (Top Red legged kittiwake)
Aircraft: Yu- 12
33%
P-78.64%
Red legaze kittwakke
Failure cases (Top Pigeon guillemot)
PPs 344 0996
Pe1 11.59%
Success cases (Top Pigeon guillemat)
Stanford Car class 138
Pigeon griillento
P484.82%
P=94 32%
Figure 7: Unveiling the feature selection behaviors of CNN. The top Iwo rows illustrate two bird classes, Red-legged Kittiwake and Pi-geon guillemot. Despite its name, the Red-legged Kittiwake were consistently highlighted on faces while the Pigeon guil-lemot was highlighted on the red legs
Figure 8: Visual explanations on Aircraft and Stanford Car benchmarks.
Evaluation on target localization.
We performed quantitative evaluations the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) (Deng et al. 2009). As a metric, we employed the pointing game (PT) presented in the study [Ribeiro al.j 2016) The PT purely measures the spatial selectiveness of the continuous visual saliency map. In the evaluation, the PT detects the maximum intensity point on the saliency map, and Hit is recorded if the maximum point is in the ground-truth annotation; otherwise, Miss is recorded. The accuracy is calculated using Ace Hit- Because multiple maximum points often arise, we em-ployed the threshold value T > 0.95 to generate binary blobs, and then we used the centroid of the biggest blob as the localization point. We empirically set the scale parameters o,=5 and =3, with spatia!=6.
fitt
Table I: PT scores according to the number of search points
Search points k
12 12
15 15
22 22
25 25
30
30
PT score
0.905
0.912
0.914
0.913
0.913
Time(s)
0.398
0.652
1.435
1.909
3.95
Table 2: Quantitative comparisons with existing models using the PT on the ILSVRC validation data using ResNet50
Methodn
PT
PT-small
Time(s)
Grads (Simonyan et al. 2013)
0.773
0.604
0.128
Deconv (Zeiler al. 2014)
0.750
0.584
0.117
Grad-CAM (Selvaraju et al. 2017)
0,901
0.754
0.183
Deconv (Zeiler al. 2014)
0.809
0.688
2.64
LIME (Ribeiro et al. 2016)
0.766
0.645
15.11
RISE (Petisiuk al. 2018)
0.907
0.787
8.11
MASK (Fong et al. 2017)
0.841
0.711
16.38
GLAS (light)
0.889
0.782
0.328
GLAS (shadow)
0.877
0.751
0.328
GLAS (fusion)
0.912
0.792
0.612

---

Under review as a conference paper at ICLR 2020
Input
MASK
Grad-CAM
RISE
GLAS
Werlon
Impala
Bhld engle
Coucal
Yorkcabire cntter
Coonhound
Input
(from the left to right Iteration 1,2,
Fusion map
Recursive process. This algorithm dis- covers the relevant parts given class order of significance. We used the adaptive stop condi- tion mentioned in Section 3.
Figure 10:
Figure 9: Visual comparison of the class-
discriminative capability of MASK, Grad-CAM, RISE, and GLAS by varying the object classes.
Input
man
rides
bicycle
trail
Input
dog
jumping
hurdle
rides bicycle the trail
A dog is jamping over hurdle
Figure 11: Visual explanation examples produced by GLAS for the image captioning model.
Table shows the execution time of GLAS according the number of grid search points. The result tends show favorable PT scores as the number of the search points grows. It starts to become saturated after & 15 in terms of performance,
the execution time continues to grow. Table 2 Dillustrates the results of comparisons with the existing methods on the ILSVRC validation dataset. GLAS achieves the best result in terms of PT score. GLAS 13 times faster than RISE even with the higher PT score. This because a considerable number of perturbed images using a randomized masking process are necessary for reliable visual explanation RISE. When RISE forced to use 450 (225 x 2) perturbed images, identical to the number used by GLAS, we observed that the PT score of RISE drops from 0.907 to 0.869. This observation tells us that GLAS perturbs and localizes the important features efficiently. We separately evaluated the cases in which the object is small in the PT-small column o Like related study (Cao al., 2015). we consider an object be small if the total area of the bounding box of the given class is smaller than one quarter of the size of the image. Even though all models encountered performance drop, GLAS still beats other models.
In our work, GLAS operations can be used together or independently. The results the last three rows of Table 2 show an ablation study on GLAS by measuring model performance with either Gaussian lighting or shadowing suppressed. [2 shows the performance the ablated GLAS comparable to that the state-of-the- art methods, outperforming all methods except Grad-CAM, RISE, and fused GLAS. Figure provides a visual comparison
the methods. GLAS and Grad- CAM clearly highlight the important region related a given class, whereas MASK and RISE suffer from nontrivial local noise. The advantage of GLAS is obvious without the noise and produces a visualization map that is highly interpretable. Because the GLAS map consists of Gaussian mixture clues, identifies the most important area without being distracted by meaningless clues.
Multiple evidence discovery by the recursive process.
Table 3 illustrates the mean intersection over union (IOU) scores of the proposed recursive process. Because the visual saliency map consists of continuous intensity values, the mean IOU scores were measured with varying thresholds, from 0 to 1.0. particular, significant progress occurs at the second iteration. Higher IOU scores over the threshold values indicate that the object regions
8

---

image instance. A deeper theo-retical Under review as a conference paper at ICLR 2020
Table 3: IOU scores of the proposed method the ILSVRC validation data the iteration in- creases, "A" represents the final iteration under the adaptive stop condition
Iteration
Threshold
0.1
0.3
0.5
0.7
0.9
1
0.53
0.51
0.40
0.31
0.09
2
0.54
0.56
0.51
0.39
0.07
3
0.55
0.56
0.52
0.37
0.06
A
0.55
0.59
0.55
0.38
0.06
uniformly highlighted. In the fusion results shown in the last column of Figure 10 multiple instances and multiple evidences are well discovered.
that
Visualization of the image captioning model.
Image captioning a challenging task for which both computer vision and natural language process- ing techniques should be considered. We constructed the image captioning model based publicly available implementations for which the fine-tuned Inception V3-based image and long short-term memory-based language models considered. Figure IT shows some visual explanation results from the image captioning model to demonstrate the applicability of GLAS. GLAS shows the ca- pability localize visual concepts such objects (man, bicycle, ball, boy, hurdle, and dog) and actions (riding, playing, and jumping).
CONCLUSION
In this study, we proposed a visual explanation method called GLAS for the black-box model. Our method is in-spired by the natural light and shadow phenomena and provides a simple yet robust way to perturb an input in-stance. The GLAS presented the ability of fine-level visual explanation at various scales through
adjustment of the Gaussian scale. Additionally, we showed the broad applicability of GLAS to various tasks. In experiments, GLAS showed state-of-the-arr performance and efficient computing time. For a future work, we plan to improve the heat map by optimizing the scale parameters of the Gauss-ian mask adaptively analysis of GLAS is also needed.
REFERENCES
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Claus-Robers Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wiss relevance propagation. PLOS ONE,
303010,, 2015.
Chunshui Cao, Xianming Liu, Yi Yang, Yinan Yu, Jiang Wang, Zilei Wang, Yongzhen Huang, Liang Wang, Chang Huang, Wei Xu, Deva Ramanan, and S. Huang Thomas. Look and think twice: Capturing top-down visual attention with feedback convolutional neural networks. In ICCV, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Yinpeng Dong, Hang Su, Jun Zhu, and Bo Zhang. Improving interpretability of deep neural net- works with semantic information. CVPR. 2017.
Ruth C. Fong and Andrea Vedaldi. Interpretable explanation black-boxes by meaningful pertur- bation. In ICCV, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine- grained categorization. In CVPR, 2013.

---

Under review as a conference paper at ICLR 2020
Hansang Lee, Minseok Park, and Junmo Kim. Plankton classification imbalanced large scale database via convolutional neural networks with transfer learning. In ICIP, 2016.
Xiaoxu Li, Liyun Yu, Dongliang Chang, Zhanyu Ma, and Jie Cao. Dual cross-entropy loss for small-sample fine-grained vehicle classification. IEEE Trunsactions on Vehicular Technology. 68:4204 2019.
Zhongqi Lin, Shaomin Mu, Feng Huang. Khattak Abdul Mateen, Minjuan Wang, Wanlin Gao, and Jingdun Jia. A unified matrix-based convolutional neural network for fine-grained image classification of wheat leaf diseases. JEEE Access, 2019.
Yann LuCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature. 521:436-444 4444 2015.
Aravindh Mahendran and Andrea Vedaldi. Salient deconvolutional networks. In ECCV, 2016.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained
visual classification of aircraft. arXiv preprint arXivv 2013.
Kanghan Oh, Myungeun Lee, Yura Lee, and Soohyung Kim. Salient object detection using recursive regional feature clustering. Information Sciences, 387:1-18, 2017.
Deepak Pathak, Philipp Krähenbühl, and Trevor Darrell. Constrained convolutional neural networks for weakly supervised segmentation. In ICCV, 2015.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. In BMVC, 2018.
Chenchen Qiu, Shaoyong Zhang, Chao Wang. Zhibin Yu, Haiyong Zheng, and Bing Zheng, Im- proving transfer learning and squeeze- and-excitation networks for small-scale fine-grained fish image classification. IEEE Access, 2018.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In ACM SIGKDD, 2016.
Šikonja Marko Robnik and Igor Kononenko. Explaining classifications for individual instances.
IEEE Transactions on Knowledge and Data Engineering, 20:589-600, 600, 2008
Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Rohert Müller. Evaluating the visualization of what a deep neural network has Iearned. IEEE Transac- tions on Neural Networks and Learning Systems, 28:2660-2673, 2016.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR, 2015.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In ICCV, 2017.
Dasom Sco, Kanghan Oh, and II-Seok Oh. Regional multi-scale approach for visually pleasing explanations of deep neural. arXiv preprint (xXiv:8807.1777 2018.
Weiwei. Shi, Yihong Gong, Xiaoyu Tao, De Cheng, and Nanning Zheng. Fine-grained image clas- sification using modified denns trained by cascaded softmax and generalized large-margin losses. IEEE Transactions on Neural Networks and Learning Systems, 30:683-694, 2018.
Karen Simonyan. Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR, 2013.
Richard Sipes and Dan Li. Using convolutional neural networks for automated fine-grained image classification of acute lymphoblastic leukemia. ICCIA. 2018.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: the all convolutional net. arXiv preprint arXiv: 1412.6806, 2014.
10

---

Under review as a conference paper at ICLR 2020
Yi Sun, Yuheng Chen, Xiaoou Wang, and Xiaogang Tang. Deep learning face representation by joint centification-veriiiaatii arXiv preprint 47X7v 2014.
Shixin Tian and Ying Cai. Visualizing deep neural networks with interaction of super-pixels. In CIKM, 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML. 2015.
Matthew Zeiler and Rob Fergus. Visualizing and understanding convolttioaal networks. In ECCV, 2013.
Jianming Zhang, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. ECCV, 2016.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, 2016.
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. In ICLR, 2017.

---

Under review as a conference paper at ICLR 2020
6 APPENDIX
A. Visualization of neuroimaging classification model.
We consider neuroimaging classification problem to show applicability of GLAS in medi- cal imaging domain. We employed 3D-magnetic resonance imaging (MRI) scans reflecting 199 Alzheimer's Disease (AD) 230 healthy Normal Control (NC) from the Alzheimer's Disease Neuroimaging Initiative (ADNI), which is publicly available. A 3D-CNN was employed for clas- sification of AD vs. NC. Due to limited dataset size, 3D-MRIs are spatially normalized based on template brain image and unsupervised learning technique (Convolutional auto-encoder) is applied before supervised learning. The overall architecture comprised of three 333 Conv layers with 10 filters each, two FC layers with 32 and 16 nodes each, and softmax activation; each of Conv layers followed by ReLU and 222 max- max-pooling: Gaussian dropout with a dropping ratio 0.8 is applied in between Conv layers; in the FC layers, we used SELU activation for speeding up learning and taking normalization effect in the FC layers. 5-fold cross validations were conducted to evaluate the classification model. The mean accuracy was 85.3
Since the Automated Anatomical Labeling (AAL) [5] map exists, all MRIs are spatially normalized based on AAL and then we considered the centroid of AAL the search points. this work, 3D- GLAS method were considered to perturb 3D-MRI data, and we empirically set a=10 and r=15. The validation MRIs of AD category are fed to 3D-GLAS, then the entire saliency maps of validation MRIs were linearly integrated and normalized. For this reason, the highlighted biomarkers have statistical significance. Since the centroids of each AAL segment are considered as search points, mark out impact of each biomarker to its corresponding AAL segment directly. In Figure 25 the hippocampus, amygdala and temporal inf were selected as the important biomarkers for the accurate classification of AD. These biomarkers have previously been known be closely related with dementia in many existing studies [1][2][3][4] In particular, the hippocampus, a brain region for learning and memory, is one the first brain biomarker, which affected by AD and undergoes severe structural changes as the disease progresses [4]
[1] R. Casanova, C.T. Whitlow, B. Wagner, J. Williamson, S.A. Shumaker,J.A Maldjian M.A. Espeland. High Dimensional Classification of Structural MRI Alzheimer's Disease Data Based on Large Scale Regularization. Frontiers in Neuroimformatics, 101:0.3389/mnnnf22011 1.00022. 2011.
[2] H. Kilian, T. Vinh-Thong, C. Gwenaelle, T. Thomas, V.M. Jose C. Pierrick. Multimodal Hippocampal Subfield Grading for Alzheimer's Disease Classification. doi:
https//doi.orr//I 10. www../////...01//31111.11 101129311 2018.
[3] S.W. Seo, N. Ayakta, L.T. Grinberg, S. Villeneuve, M. Lehmann, B. Reed G.D. Rabinovici. Regional correlations between [11C] PIB PET and post-mortem burden of amyloid-beta pathology in diverse neuropathological cohort. Neuroimage Clinic 13 (Suppl. C), 130-137,2017.
[4] S. Tian, C Ying. Visualizing Deep Networks with Interaction of Super-pixel with Interaction of Super-pixels. In CIKM, 2017.
N. N. Tzourio-Mazoyer B. Landeau, D. Papathanassiou, F. Crivello, O. Etard, N. Deleroix, Bernard Mazoyer M. Joliot, Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. NeuroImage 15: 273-289, 2002
Z=-50
Z=-45
Z=-40
Z=-35
Z=-30
Figure 12: Visual distribution of discriminative biomarkers the classification of AD. The first rows illustrate brain template image overlapped with important biomarkers (rank 1 of Table 1: red, rank 2: green, rank 3: blue, rank 4: yellow, rank 5: purple). The second rows show AAL. wise saliency map.
12

---

Under review as a conference paper at ICLR 2020
B. Salient explanation results for fine-grained classification task.
In this part, we provide numerous visual examples to show the precision explanation. We chose 13 bird categories CUB200 that have the unique characteristics. the experiment, we used the same sigma value of 3.0.
Red winged blackbird
Figure 13: Red winged blackbird. The uniform characteristic red shoulder patch.
Northern flicker
Figure 14: Red winged blackbird. The uniform characteristic is red shoulder patch.
13

---

Under review as a conference paper at ICLR 2020
Figure 15: Crested auklet. The uniform characteristic orange beak.
Figure 16: Eared grebe. The uniform characteristic red eyes.
14

---

Under review as a conference paper at ICLR 2020
Figure 17: Red headed woodpecker. The uniform characteristic is red head.
Figure 18: Red bellied woodpecker. The uniform characteristic is black and white barred patterns on their back, wings and tail. Adult males have a red cap going from the bill to the nape; females have red patch the nape and another above the bill.
15

---

Under review as a conference paper at ICLR 2020
Figure 19: Pileated woodpecker. The uniform characteristic is red crest.
Figure 20: Prairie warbler. The uniform characteristic is dark streaks on the flanks and head.
16

---

Under review as a conference paper at ICLR 2020
Figure 21: Myrtle warbler. The uniform characteristics are yellow crown, rump and flank patches.
Figure 22: European goldfinch. The uniform characteristics are red face and broad yellow bar on the wings.
17

---

Under review as a conference paper at ICLR 2020
ARIEOO
Figure 23: Gold Winged Warbler. The uniform characteristics are yellow crown and wing patch.
Figure 24: Rose breasted grosbeak. The uniform characteristic is bright red-red patch on the breast.
18

---

Under review as a conference paper at ICLR 2020
Figure 25: Red faced cormorant. The uniform characteristic is red patch around the eyes.
19


Reviews:
[{"review_id": "Ske5I-NPFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a simple procedure to display the salient areas that determine classification decisions from deep networks. In practice, these area are computed by shadowing the image using a Gaussian and measuring the network contribution at every location of the image. Results in terms of \"Pointing game\" metric (that measure similarity between computed saliency result and ground truth) are provided on the Imagenet validation dataset. \n\n1 Positive aspects:\n- Experiments are conducted on many datasets and show visually convincing results.  \n- state-of-the-art on ImageNet compared to the RICE approach. \n\n2 Negative aspects: \n- Very simple approach\n- Depends on many parameters (t1, t2, sigma_s, sigmal_l,  sigma_spatial) \n- Improvement of PT w.r.t. Rice is not very high: 0.912 compared to 0.907.The variants Light, shadow, fusion approaches are not described in the text.  \n- The paper is badly written. \n-- Many sentences are uninformative (e.g. abstract :  \"We prove the effectiveness of GLAS for fine-grained classification using the fine-grained classification dataset\")\n-- There is copy-pasted text everywhere (e.g. fea - tures , formu - lated, etc.), showing there has not been proof-reading of the paper, which is pretty disconcerting to reviewers. \n- There are multiple typos including misspelled names in the bib.\n- blurry figures.  \n\nMy initial rating is mainly motivated by the minor experimental improvement wrt RICE that is not clearly explained. There would also be a lot of work for improving the writing of the paper.   \n\nMinor : lambda is 10^-5, not 10^5 ? \nis sigma_s used in the algo or any equation? Where is it defined? ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkeo4WgCYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a new method in the family of local perturbation-based interpretations for deep networks and more specifically for fine-grained classification tasks. Compared to other saliency map methods (gradient-based, propagation-based, etc), this family has the advantage of needing only black-box access to the model. The introduced method GLAS, scans over an image and lights/shadows each part of the image to assign an importance score to different regions of the image based on the change in model's prediction. The motivation of this work is to increase the inherently low speed of (some) methods in this family and to give better explanations by \n\n\n\nI vote for rejecting this paper as the contributions to what already exists in the literature are not clear and the provided experimental results are not convincing.\n\nCompared to previous perturbation-based methods, the main advantage seems to be speed. There are perturbation-based methods that do not suffer from low speed e.g. Dabkowski & Gal and give real-time perturbation-based saliency maps. The authors do not mention this work (and similar works) and do not compare both their speed and their performance against it.\n\nOne important problem (probably the most important) with the perturbation-based saliency maps is the fact that the perturbations might push a given image out of the true data manifold and therefore give an invalid interpretation of the model. The authors do not discuss the matter and how their method would address this issue. Intuitively, the introduced algorithm, more specifically the RGLAS algorithm, seems to suffer from this issue not any less than other existing methods.\n\nThe experimental results seek to demonstrate the superiority of the introduced method over rival methods. The justification behind the provided visual examples is their focus on more discriminative features (in human eyes). This does not necessarily mean that a given saliency map is more \"truceful\"; i.e. having a more visually appealing saliency map has nothing to do with a more truthful explanation of a model's decision making. The results focused on the mistakes of the model seem more convincing and interesting.\n\n The objective results first focus on the target localization metric which has traditionally been used in the literature. Although it is much faster to execute, the introduced method is only marginally superior to other methods. The most important problem, however, is that as mentioned above, there are fast methods in the literature and therefore a fair objective comparison is should include other methods as well. Secondly, the IOU measure is used. GLAS is not compared to other methods in this metric.\n\nThe authors mention the effectiveness of their work for \"fine-grained\" classification tasks while throughout the paper there is no convincing evidence or discussion that the method is curated for such tasks. As mentioned above, changing the scale parameter for getting more visuall appealing saliency maps is not enough evidence.\n\nAll in all, the true contribution of this work to other existing methods in this family is not enough for this venue.\n\n\nA few questions and suggestions:\n* How should one adjust the scale parameter? In other words, what is the hyper-parameter search scheme for this method which would make it robust against the human-biased choice of hyper-parameters which would result in visually more appealing saliency maps but not necessarily explain the model?\n* Explanation of RGLAS is not clear.\n* For a general reader, metrics such as IOU should be explained more clearly.\n* The paper has many many typing errors.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeZgnsWcr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a simple and effective method for pinpointing salient features contributing to discriminating different classes in classification. It is based on masking images using Gaussian Gaussian light and shadow (GLAS) and estimating its impact on output. The authors also develop an iterative method to identify multiple instances. Experiments quantitatively evaluate the proposed method on pointing game using ILSVRC validation set, where the proposed method outperforms recent related methods. \n\nThe technical novelty of this paper is marginal and the experimental evaluation is not convincing. \n \n1) Marginal novelty compared to related work\nThere have been several masking-based black-box methods. Considering the method of RISE (Petsiuk et al., BMVC19) in particular, the novelty of the proposed method is marginal; the proposed method uses Gaussian light and shadow at individual positions while RISE uses random masks. \n\n2) Hyperparameter sensitivity \nThe proposed method includes several, at least four, hyperparameters to be tuned: T, sigma_l, sigma_ss, sigma_spatial. These may affect the results significantly, but the experimental section does not discuss their sensitivity. Are they robust across different datasets? It is not clear from the experiments. \n\n3) Experimental comparison\nThe comparisons are not convincing. The results on fine-grained classification are presented qualitatively, and the quantitative comparison is done only on a single dataset, ILSVRC validation set, which is very limited considering experiments in the related work, e.g., three benchmark experiments of RISE RISE (Petsiuk et al., BMVC19) on PASCAL VOC, MSCOCO, and ImageNet. Furthermore, in the ILSVRC experiments, the gap from Grad-CAM and RISE is not significant in terms of accuracy. Considering several hyperparameters of the proposed method to be tuned, these results appear less appealing. I hope the authors provide more convincing experiments, e.g., on the same benchmarks of RISE.  ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is interested in finding salient areas in a deep learning image classification setting. The introduced method relies on masking images using Gaussian Gaussian light and shadow (GLAS) and estimating its impact on output.

As noted by all reviewers, the paper is too weak for publication in its current form:
- Novelty is very low.
- Experimental section not convincing enough, in particular some metrics are missing.
- The writing should be improved.
Decision: Reject