Year: 2020

Abstract:
Saliency maps that identify the most informative regions of an image for a classifier are valuable for model interpretability. A common approach to creating saliency maps involves generating input masks that mask out portions of an image to maximally deteriorate classification performance, or mask in an image to preserve classification performance. Many variants of this approach have been proposed in the literature, such as counterfactual generation and optimizing over a Gumbel-Softmax distribution. Using a general formulation of masking-based saliency methods, we conduct an extensive evaluation study of a number of recently proposed variants to understand which elements of these methods meaningfully improve performance. Surprisingly, we find that a well-tuned, relatively simple formulation of a masking-based saliency model outperforms many more complex approaches. We find that the most important ingredients for high quality saliency map generation are (1) using both masked-in and masked-out objectives and (2) training the classifier alongside the masking model. Strikingly, we show that a masking model can be trained with as few as 10 examples per class and still generate saliency maps with only a 0.7-point increase in localization error.

heading: 1 INTRODUCTION
text: The success of CNNs (Krizhevsky et al., 2012; Szegedy et al., 2015; He et al., 2016; Tan & Le, 2019) has prompted interest in improving understanding of how these models make their predictions. Particularly in applications such as medical diagnosis, having models explain their predictions can improve trust in them. The main line of work concerning model interpretability has focused on the creation of saliency maps–overlays to an input image that highlight regions most salient to the model in making its predictions. Among these, the most prominent are gradient-based methods (Simonyan et al., 2013; Sundararajan et al., 2017; Selvaraju et al., 2018) and masking-based methods (Fong & Vedaldi, 2017; Dabkowski & Gal, 2017; Fong & Vedaldi, 2018; Petsiuk et al., 2018; Chang et al., 2019; Zintgraf et al., 2017). In recent years, we have witnessed an explosion of research based on these two directions. With a variety of approaches being proposed, framed and evaluated in different ways, it has become difficult to assess and fairly evaluate their additive contributions.
In this work, we investigate the class of masking-based saliency methods, where we train a masking model to generate saliency maps based on an explicit optimization objective. Using a general formulation, we iteratively evaluate the extent to which recently proposed ideas in the literature improve performance. In addition to evaluating our models against the commonly used Weakly Supervised Object Localization (WSOL) metrics, the Saliency Metric (SM), and the more recently introduced Pixel Average Precision (PxAP; Choe et al., 2020), we also test our final models against a suite of “sanity checks” for saliency methods (Adebayo et al., 2018; Hooker et al., 2018).
Concretely, we make four major contributions. (1) We find that incorporating both masked-in classification maximization and masked-out entropy maximization objectives leads to the best saliency maps, and continually training the classifier improves the quality of generated maps. (2) We find that the masking model requires only the top layers of the classifier to effectively generate saliency maps. (3) Our final model outperforms other masking-based methods on WSOL and PxAP metrics. (4) We find that a small number of examples—as few as ten per class—is sufficient to train a masker to within the ballpark of our best performing model.

heading: 2 RELATED WORK
text: Interpretability of machine learning models has been an ongoing topic of research (Ribeiro et al., 2016; Doshi-Velez & Kim, 2017; Samek et al., 2017; Lundberg et al., 2018). In this work, we focus on interpretability methods that involve generating saliency maps for image classification models. An overwhelming majority of the methods for generating saliency maps for image classifiers can be assigned to two broad families: gradient-based methods and masking-based methods.
Gradient-based methods, such as using backpropagated gradients (Simonyan et al., 2013), Guided Backprop (Springenberg et al., 2015), Integrated Gradients (Sundararajan et al., 2017), GradCam (Selvaraju et al., 2018), SmoothGrad (Smilkov et al., 2017) and many more, directly use the backpropagated gradients through the classifier to the input to generate saliency maps.
Masking-based methods modify input images to alter the classifier behavior and use the regions of modifications as the saliency map. Within this class of methods, one line of work focuses on optimizing over the masks directly: Fong & Vedaldi (2017) optimize over a perturbation mask for an image, Petsiuk et al. (2018) aggregates over randomly sampled masks, Fong & Vedaldi (2018) performs an extensive search for masks of a given size, while Chang et al. (2019) includes a counterfactual mask-infilling model to make the masking objective more challenging. The other line of work trains a separate masking model to produce saliency maps: Dabkowski & Gal (2017) trains a model that optimizes similar objectives to Fong & Vedaldi (2017), Zolna et al. (2020) use a continually trained pool of classifiers and an adversarial masker to generate model-agnostic saliency maps, while Fan et al. (2017) identifies super-pixels from the image and then trains the masker similarly in an adversarial manner.
Salient Object Detection (Borji et al., 2014; Wang et al., 2019) is a related line of work that concerns identifying salient objects within an image as an end in itself, and not for the purpose of model interpretability. While it is not uncommon for these methods to incorporate a pretrained image classification model to extract learned visual features, they often also incorporate techniques for improving the quality of saliency maps that are orthogonal to model interpretability. Salient object detection methods that are trained on only image-level labels bear the closest similarity to saliency map generation methods for model interpretability. Hsu et al. (2017) and follow-up Hsu et al. (2019) train a masking model to confuse a binary image-classification model that predicts whether an image contains an object or is a ‘background’ image. Wang et al. (2017) apply a smooth pooling operation
and a Foreground Inference Network (a masking model) while training an image classifier to generate saliency maps as a secondary output.
Evaluation of saliency maps The wave of saliency map research has also ignited research on evaluation methods for these saliency maps as model explanations. Adebayo et al. (2018) and Hooker et al. (2018) propose sanity checks and benchmarks for the saliency maps. Choe et al. (2020) propose Pixel Average Precision (PxAP), a pixel-wise metric for scoring saliency maps that accounts for mask binarization thresholds, while Yang & Kim (2019) create a set of metrics as well as artificial datasets interleaving foreground and background objects for evaluating the saliency maps. These works have shown that a number of gradient-based methods fail the sanity checks or perform no better than simple edge detectors. Hence, we choose to focus on masking-based methods in this paper.

heading: 3 MASKING-BASED SALIENCY MAP METHODS
text: We start by building a general formulation of masking-based saliency map methods. We take as given a trained image classifier F : x → y, that maps from image inputs x ∈ RH×W×C to class predictions ŷ ∈ [0, 1]K , evaluated against ground-truth y ∈ {1 · · ·K}. Our goal is to generate a mask m ∈ [0, 1]H×W for each image x such that the masked-in image x m or the masked-out image x (1−m) maximizes some objective based on output of a classifier given the modified image. For instance, we could attempt to mask out parts of the image to maximally deteriorate the classifier’s performance. This mask m then serves as a saliency map for the image x. Concretely, the per-image objective can be expressed as:
arg min m
λoutLout ( F (x (1−m); θF ), y ) + λinLin ( F (x m; θF ), y ) +R(m),
where Lout, Lin are the masked-out and masked-in objectives over the classifier output, λout, λin are hyperparameters controlling weighting of these two objectives, θF the classifier parameters, and R(m) a regularization term over the mask. The masked-in and masked-out losses, Lout and Lin, correspond to finding the smallest destroying region and smallest sufficient region as described in Dabkowski & Gal (2017). Candidates for Lout include negative classification cross-entropy and prediction entropy. For Lin, the obvious candidate is the classification cross-entropy of the masked-in image. We set λin = 0 or λout = 0 if we only have either a masked-in or masked-out objective.
The above formulation subsumes a number of masking-based methods, such as Fong & Vedaldi (2017); Dabkowski & Gal (2017); Zolna et al. (2020). We follow Dabkowski & Gal, amortize the optimization by training a neural network masker M : x→ m, and solve for:
arg min θM
λoutLout ( F (x (1−M(x; θM )); θF ), y ) +λinLin ( F (x M(x; θM ); θF ), y ) +R(M(x; θM )),
where M is the masking model and θM its parameters. In our formulation, we do not provide the masker with the ground-truth label, which differs from certain other masking-based saliency works (Dabkowski & Gal, 2017; Chang et al., 2019; Fong & Vedaldi, 2018). In practice, we often desire model explanations without the availability of ground-truth information, so we focus our investigation on methods that require only an image as input.

heading: 3.1 MASKER ARCHITECTURE
text: We use a similar architecture to Dabkowski & Gal and Zolna et al.. The masker takes as input activations across different layers of the classifier, meaning it has access to the internal representation of the classifier for each image. Each layer of activations is fed through a convolutional layer and upsampled (with nearest neighbor interpolation) so they all share the same spatial resolution. All transformed layers are then concatenated and fed through another convolutional layer, upsampled, and put through a sigmoid operation to obtain a mask of the same resolution as the input image. In all our experiments, we use a ResNet-50 (He et al., 2016) as our classifier, and the masker has access to the outputs of the five major ResNet blocks. Figure 1B shows the architecture of our models.
Following prior work (Fong & Vedaldi, 2017), we apply regularization on the generated masks to avoid trivial solutions such as masking the entire image. We apply L1 regularization to limit the size of masks and Total Variation (TV) to encourage smoothness. Details can be found in Appendix A.1.

heading: Model OM ↓ LE ↓ SM ↓ PxAP ↑

heading: Train-Validation Set

heading: Validation Set

heading: 3.2 CONTINUAL TRAINING OF THE CLASSIFIER
text: Because neural networks are susceptible to adversarial perturbations (Goodfellow et al., 2015), masking models can learn to perturb an input to maximize the above objectives for a given fixed classifier without producing intuitive saliency maps. While directly regularizing the masks is one potential remedy, Zolna et al. (2020) propose to train the masker against a diverse set of classifiers. In practice, they simulate this by continually training the classifier on masked images, retain a pool of past model checkpoints, and sample from the pool when training the masker.
We adopt their approach and distinguish between a masker trained against a fixed classifier (FIX) and against a pool of continually trained classifiers (CA, for Classifier-Agnostic). We highlight that saliency maps for FIX and CA address fundamentally different notions of saliency. Whereas a FIX approach seeks a saliency map that explains what regions are most salient to a given classifier, a CA approach tries to identify all possible salient regions for any hypothetical classifier (hence, classifier-agnostic). In other words, a CA approach may be inadequate for interpreting a specific classifier and is better suited for identifying salient regions for a class of image classification models.

heading: 4 EXPERIMENTAL SETUP
text: We perform our experiments on the official ImageNet training and validation set (Deng et al., 2009) and use bounding boxes from the ILSVRC’14 localization task. Because we perform a large number of experiments with hyperparameter search to evaluate different model components, we construct a separate held-out validation set of 50,000 examples (50 per class) from the training set with bounding box data that we use as validation for the majority of our experiments (which we refer to as our “TrainValidation” set) and use the remainder of the training set for training. For each model configuration, we train the models 5 times on different random seeds and report the mean and standard error of the results. We reserve the official validation set for the final evaluation.

heading: 4.1 EVALUATION METRICS
text: Weakly-supervised object localization task metrics (WSOL) is a common task for evaluating saliency maps. It involves generating bounding boxes for salient objects in images and scoring them against the ground-truth bounding boxes. To generate bounding boxes from our saliency maps, we binarize the saliency map based on the average mask pixel value and use the tightest bounding box around the largest connected component of our binarized saliency map. We follow the evaluation protocol in ILSVRC ’14 computing the official metric (OM), localization error (LE) and pixel-wise F1 score between the predicted and ground-truth bounding boxes.
Saliency metric (SM) proposed by Dabkowski & Gal (2017) consists of generating a bounding box from the saliency map, upsampling the region of the image within the bounding box and then evaluating the classifier accuracy on the upsampled salient region. The metric is defined as s(a, p) = log(max(a, 0.05)) − log(p), where a is the size of the bounding box, and p is the probability the classifier assigns to the true class. This metric can be seen as measuring masked-in and upsampled classification accuracy with a penalty for the mask size. We use the same bounding boxes as described in WSOL for consistency.
Pixel Average Precision (PxAP) proposed by Choe et al. (2020) scores the pixel-wise masks against the ground-truth masks and computes the area under the precision-recall curve. This metric is computed over mask pixels rather than bounding boxes and removes the need to threshold and binarize the mask pixels. PxAP is computed over the OpenImages dataset (Benenson et al., 2019) rather than ImageNet because it requires pixel-level ground-truth masks.

heading: 5 EVALUATION OF SALIENCY MAP METHODS
text: To determine what factors and methods contribute to improved saliency maps, we perform a series of evaluation experiments in a cascading fashion. We isolate and vary one design choice at a time, and use the optimal configuration from one set of experiments in all subsequent experiments. Our baseline models consist of a masker trained with either a fixed classifier (FIX) or a pool of continually trained classifiers (CA). As WSOL is the most common task for evaluating saliency maps, we use LE as the metric for determining the ‘best’ model for model selection. We show our model scores across experiments in Table 3. Each horizon block represents a set of experiments varying one design choice. The top half of the table is evaluated on our Train-Validation split, while the bottom half is evaluated on the validation data from ILSVRC ’14.
Masking Objectives (Rows a–f) We first consider varying the masker’s training objective, using only one objective at a time. We use the three candidate objectives described in Section 3: maximizing masked-in accuracy, minimizing masked-out accuracy and maximizing masked-out entropy. For a masked-out objective, we set λout = 1, λin = 0, and the opposite for masked-in objectives. For each configuration, we perform a random hyperparameter search over the L1 mask regularization coefficients λM and λTV as well as the learning rate and report results from the best configuration from the Train-Validation set. More details on hyperparameter choices can be found in Table 2.
Consistent with Zolna et al. (2020), we find that training the classifier along with the masker improves the masker, with CA models generally outperforming FIX models, particularly for the WSOL metrics. However, the classification-maximization FIX model still performs comparably with its CA counterpart and in fact performs best overall when measured by SM given the similarity between the training objective and the second term of the SM metric. Among the CA models, entropymaximization and classification-minimization perform the best, while the classification-maximization
objective performs worst. On the other hand, both mask-out objectives perform extremely poorly for a fixed classifier. We show how different masking objectives affect saliency map generation in Figure 2.
European fire salamander
Input
Appenzeller
Input
FIX MaxClass (I) FIX MinClass (O) FIX MaxEnt (O) CA MaxClass (I) CA MinClass (O) CA MaxEnt (O)
FIX MinClass (O) +MaxClass (I) FIX MaxEnt (O) +MaxClass (I) CA MinClass (O) +MaxClass (I) CA MaxEnt (O) +MaxClass (I)
Combining Masking Objectives (Rows g–j) Next, we combine both masked-in and masked-out objective during training, setting λout = λin = 0.5. Among the dual-objective models, entropymaximization still outperforms classification-minimization as a masked-out objective. Combining both masked-in classification-maximization and masked-out entropy-maximization performs best for both FIX and CA models, consistent with Dabkowski & Gal (2017). From our hyperparameter search, we also find that separately tuning λM,in and λM,out is highly beneficial (see Table 2). We use the classification-maximization and entropy-maximization dual objectives for both FIX (Row g) and CA (Row i) models in subsequent experiments.
Varying Observed Classifier Layers (Rows k–r) We now vary which hidden layers of the classifier the masker has access to. As described in Section 3.1, the masking model has access to hidden activations from five different layers of a ResNet-50 classifier. To identify the contribution of information from each layer to the masking model, we train completely new masking models with access to only a subset of the classifier layers. We number the layers from 1 to 5, with 1 being the earliest layer with the highest resolution (56× 56) and 5 being the latest (7× 7). We show a relevant subset of the results from varying the observed classifier layers in Table 3. The full results can be found in the Table 4 and we show examples of the generated masks in Figure 3.
Masking models with access to activations of later layers starkly outperform those using activations from earlier layers. Whereas the Layer[3], Layer[4] and Layer[5] models are still effective, the Layer[1] and Layer[2] models tend to perform poorly. Similarly, we find that the best cascading combination of layers is layers 4 and 5 CA models, and 3–5 for FIX models (Rows n, r), slightly but consistently outperforming the above models with all layers available to the masker. This suggests that most of the information relevant for generating saliency maps is likely contained within the later layers. For simplicity, we use only classifier layers 4 and 5 for subsequent experiments.
Counterfactual Infilling and Binary Masks (Rows s–z) Chang et al. (2019) proposed generating saliency maps by learning a Bernoulli distribution per masking pixel and additionally incorporating a counterfactual infiller. Agarwal & Nguyen (2019) similarly uses an infiller when producing saliency maps. First, we consider applying counterfactual infilling to the masked images before feeding them to the classifier. The modified inputs are Infill(X (1 −m), (1 −m)) and Infill(X m,m) for

heading: Radiator
text: masked-out and masked-in infilling respectively, where Infill is the infilling function that takes as input the masked input as well as the mask. We consider three infillers: the Contextual Attention GAN (Yu et al., 2018) as used in Chang et al.1, DFNet (Hong et al., 2019), and a Gaussian blur infiller as used in Fong & Vedaldi (2018). Both neural infillers are pretrained and frozen.
For each infilling model, we also train a model variant that outputs a discrete binary mask by means of a Gumbel-Softmax layer (Jang et al., 2017; Maddison et al., 2017). We experiment with both soft and hard (Straight-Through) Gumbel estimators, and temperatures of {0.01, 0.05, 0.1, 0.5}. We show a relevant subset of the results in the fourth and fifth blocks of Table 3, and examples in Figure 6 and Figure 7. We do not find major improvements from incorporating infilling or discrete masking based on WSOL metrics, although we do find improvements from using the DFN infiller for SM. Particularly for CA, because the classifier is continually trained to classify masked images, it is able to learn to both classify unnaturally masked images as well as to perform classification based on masked-out evidence. As a result, the benefits of incorporating the infiller may be diminished.

heading: 5.1 EVALUATION ON VALIDATION SET
text: Based on the above, we identify a simple recipe for a good saliency map generation model: (1) use both masked-in classification maximization and masked-out entropy maximization objectives, (2) use only the later layers of the classifier as input to the masker, and (3) continually train the classifier. To validate the effectiveness of this simple setup, we train a new pair of FIX and CA models based on this configuration on the full training set and evaluate on the actual ILSVRC ’14 validation set. We compare the results to other models in the literature in the bottom block of Table 3. Consistent with above, the CA model outperforms the FIX model. It also outperforms other saliency map extraction methods on WSOL metrics and PxAP. We highlight that some models we compare to (Rows E, G) are provided with the ground-truth target class, whereas our models are not–this may explain the underperformance on certain metrics such as SM, which is partially based on classification accuracy.

heading: 5.2 SANITY CHECKS
text: Adebayo et al. (2018) and Hooker et al. (2018) propose “sanity checks” to verify whether saliency maps actually reflect what information classifiers use to make predictions and show that many proposed interpretability methods fail these simple tests. We apply these tests to our saliency map models to verify their efficacy. On the left of Figure 4, we show the RemOve-and-Retrain (ROaR) test proposed by Hooker et al., where we remove the top t% of pixels from training images based on our generated saliency maps and use them to train entirely new classifiers. If our saliency maps truly identify salient portions of the image, we should see large drops in classifier performance as t increases. Both FIX and CA methods pass this test, with classifier accuracy falling precipitously as we mask out more pixels. On the right of Figure 4, we perform the the Model Parameter Randomization Test (MPRT) proposed by Adebayo et al.. We randomize parameters of successive layers of the
1The publicly released CA-GAN is only trained on rectangular masks, but Chang et al. nevertheless found positive results from applying it, so we follow their practice. DFNet is trained on irregularly shaped masks.
classifier, starting from upper-most logits layer to the lowest convolutional layers, and generate saliency maps using the partially randomized classifiers. We then compute the similarity of the saliency maps generated from using the partially randomized classifier, and those using the original classifier. Our saliency maps become less similar as more layers are randomized, passing the test. The results for the Data Randomization Test (DRT) can be found in Table 5.

heading: 6 FEW-SHOT EXPERIMENTS
text: Given the relative simplicity of our best-performing saliency map models and the fact that the masker uses only the top layers of activations from the classifier, we hypothesize that learning to generate saliency maps given strong classifier is a relatively simple process.
To test this hypothesis, we run a set of experiments severely limiting the number of training steps and unique examples that the masker is trained on. The ImageNet dataset consists of 1,000 object classes, with up to 1,250 examples per class. We run a set of experiments restricting both the number of unique classes seen as well as the number of examples seen per class while training the masker. We also limit the number of training steps be equivalent to one epoch through the full training set. Given the randomness associated with subsampling the examples, we randomly subsample classes and/or examples 5 times for each configuration and compute the median score over the 5 runs. We report results on the actual validation set for ILSVRC ’14 (LE) and test set for OpenImages (PxAP). Examples of saliency maps for these models can be found in Figure 9.
We show the results for our CA model in Figure 5 and for the FIX model in Figure 8. Strikingly, we find that very few examples are actually required to train a working saliency map model. In particular, training on just 10 examples per class produces a model that gets only 0.7 LE more than using all of the training data and only 2.7 more than the fully trained model.
On the other hand, we find that the diversity of examples across classes is a bigger contributor to performance than the number of examples per class. For instance, training on 10 examples across all 1,000 classes gets an LE of 38.5, which is lower than training on 125 examples across only 100 classes. A similar pattern can be observed in the PxAP results.
Above all, these few-shot results indicate that training an effective saliency map model can be significantly simpler and more economical than previously thought. Saliency methods that require training a separate model such as Dabkowski & Gal (2017) and Zolna et al. (2020) are cheap to run at inference, but require an expensive training procedure, compared to gradient-based saliency methods or methods involving a per-example optimization. However, if training a masking model can be a lightweight procedure as we have demonstrated, then using masking models to generate saliency maps can now be a cheap and effective model interpretability technique.

heading: 7 DISCUSSION AND CONCLUSIONS
text: In this work, we systematically evaluated the additive contribution of many proposed improvements to masking-based saliency map methods. Among the methods we tested, we identified that only the following factors meaningfully contributed to improved saliency map generation: (1) using both masked-in and masked-out objectives, (2) using the later layers of the classifier as input and (3) continually training the classifier. This simple setup outperforms other methods on WSOL metrics and PxAP, and passes a suite of saliency map sanity checks.
Strikingly, we also found that very few examples are actually required to train a saliency map model, and training with just 10 examples per class can achieve close to our best performing model. In addition, our masker model architecture is extremely simple: a two-layer ConvNet. Together, this suggests that learning masking-based saliency map extraction may be simpler than expected when given access to the internal representations of a classifier. This unexpected observation should make us reconsider both the methods for extracting saliency maps and the metrics used to evaluate them.

heading: A MODEL DETAILS

heading: A.1 MASK REGULARIZATION
text: Without regularization, the masking model may learn to simply mask in or mask out the entire image, depending on the masking objective. We consider two forms of regularization. The first is L1 regularization over the mask pixels m, which directly encourages the masks to be small in aggregate. The second is Total Variation (TV) over the mask, which encourages smoothness:
TV(m) = ∑ i,j (mi,j −mi,j+1)2 + ∑ i,j (mi,j −mi+1,j)2,
where i, j are pixel indices. TV regularization was found to be crucial by Fong & Vedaldi (2017) and Dabkowski & Gal (2017) to avoid adversarial artifacts. Hence, we have:
R(m) = λM ‖m‖1 + λTVTV(m). (1)
Following Zolna et al. (2020), we only apply L1 mask regularization if we are using masked-in objective and the masked-in image is correctly classified, or we have a masked-out objective and the masked-out image is incorrectly classified–otherwise, no L1 regularization is applied for that example. In cases where we have both masked-in and masked-out objective, we have separate λM,in and λM,out regularization coefficients.

heading: A.2 CONTINUAL TRAINING OF THE CLASSIFIER
text: We largely follow the setup for training classifier-agnostic (CA) models from Zolna et al. (2020). Notably, when training on the masker objectives, we update θM but note θF , to prevent the classifier from being optimized on the masker’s objective. We maintain a pool of 30 different classifier weights in our classifier pool. We point the reader to Zolna et al. (2020) for more details.

heading: A.3 HYPERPARAMETERS
text: We show in Table 2 the space of hyperparameter search and hyperparameters for the best results, as show in the Table 1 in the main paper. We performed a random search over λM,out, λM,in, and λTV.
Aside from the hyperparameters shown in Table 2, we used a learning rate of 0.001 and a batch size of 72. We trained for 3.4 epochs (17 epochs on 20% of data) for all Train-Validation experiments and for 12 epochs for the Validation set experiments. Likewise, we use a learning rate decay of 5 for Train-Val experiments and 20 for Validation set experiments. For dual objective models, we used λout = λin − 0.5. We use the Adam optimize with 0.9 momentum and 1e-4 weight decay.

heading: Model Hyperparameters

heading: Validation Set

heading: B SUPPLEMENTARY RESULTS

heading: B.1
text: B.2 VARYING OBSERVED CLASSIFIER LAYERS
We show the full set of per-layer and layer-combination results in Table 4.

heading: B.3 COUNTERFACTUAL INFILLING AND BINARY MASKS
text: We show examples of saliency maps generated with counterfactual infillers in Figure 6 and incorporation of Gumbel-Softmax to generate masks with binary pixel values in Figure 7.

heading: B.4 SANITY CHECKS
text: We show in Table 5 the results for the Data Randomization Test (DRT) proposed by Adebayo et al. (2018). Here, we train a new classifier on the same training data but with labels shuffled across all images. We find that the similarity of saliency maps generated given a regularly trained classifier compared to those given a classifier trained on shuffled labels is low, indicating that the saliency maps reflect information learned from a well-formed image classification task.

heading: B.5 FEW-SHOT EXPERIMENTS
text: We show in Figure 8 the results for few-shot experiments using the FIX model configuration. We similarly find that very few examples are needed to train a good saliency map model.

heading: Model Rank Correl(Abs) Rank Correl(No Abs) HOGS Similarity SSIM

heading: Train-Validation Set

heading: Model OM ↓ LE ↓ F1 ↑ SM ↓ PxAP ↑ Mask
text: Red-breasted merganser
Input
Cardigan Welsh corgi
Input
Chainlink fence
Input
Gordon setter
Input
FI X
1000 classes 1 examples 100 classes 10 examples 1000 classes 10 examples 100 classes 125 examples 1000 classes 125 examples 100 classes 1250 examples 1000 classes 1250 examples Full Training
CA FI
X CA
FI X
CA FI
X CA
Figure 9: Examples of saliency maps computing from models trained with less data. Columns correspond to the models shown in Figure 5 and Figure 8. Even models trained with very few examples per class produce saliency maps similar to the fully trained model.

Review summaries:
By the first look, this work itself does not introduce any new architecture or novel algorithm. It takes what is considered as the popular choices in generating classifier saliency masks, and conducts quite extensive sets of experiments to dissect the components by their importance. The writing is pretty clear in narrative and the experimental findings are surprising and significant.