Investigating and Simplifying Masking-based Saliency Methods for Model Interpretability
Authors: Jason Phang
Jungkyu Park
Krzysztof J. Geras
Venue:  (2021)
Keywords: saliency maps
interpretability
explainable AI
image recognition
image masking
adversarial training

Abstract: Saliency maps that identify the most informative regions of an image for a classifier are valuable for model interpretability. A common approach to creating saliency maps involves generating input masks that mask out portions of an image to maximally deteriorate classification performance, or mask in an image to preserve classification performance. Many variants of this approach have been proposed in the literature, such as counterfactual generation and optimizing over a Gumbel-Softmax distribution. Using a general formulation of masking-based saliency methods, we conduct an extensive evaluation study of a number of recently proposed variants to understand which elements of these methods meaningfully improve performance. Surprisingly, we find that a well-tuned, relatively simple formulation of a masking-based saliency model outperforms many more complex approaches. We find that the most important ingredients for high quality saliency map generation are (1) using both masked-in and masked-out objectives and (2) training the classifier alongside the masking model. Strikingly, we show that a masking model can be trained with as few as 10 examples per class and still generate saliency maps with only a 0.7-point increase in localization error.

Under review as a conference paper at ICLR 2021
INVESTIGATING AND SIMPLIFYING MASKING-BASED SALIENCY MAP METHODS FOR MODEL INTER- PRETABILITY
Anonymous authors Paper under double-blind review
ABSTRACT
Saliency maps that identify the most informative regions of an image for clas- sifier are valuable for model interpretability. A common approach to creating saliency maps involves generating input masks that mask out portions an im- age to maximally deteriorate classification performance, mask in an image preserve classification performance. Many variants this approach have been proposed in the literature, such counterfactual generation and optimizing over a Gumbel-Softrax distribution. Using a general formulation of masking-basse saliency methods, we conduct extensive evaluation study of number of recently proposed variants to understand which elements of these methods meaningfully improve performance. Surprisingly, we find that well-tuned, relatively simple formulation of a masking- approaches. We find that the most important ingredients for high quality saliency map generation are (1) using both masked-in and masked-out objectives and (2) training the classifier alongside the masking model. Strikingly, we show that masking model can be trained with as few 10 examples per class and still generate saliency maps with only 0.7-point increase in localization error.
saliency model outperforms many more complex
INTRODUCTION
The success of CNNs (Krizhevsky et al., 2012; Szegedy et al., 2015; He et al., 2016; Tan & Lc, 2019) has prompted interest in improving understanding of how these models make their predictions. Particularly in applications such as medical diagnosis, having models explain their predictions can improve trust in them. The main line of work concerning model interpretability has focused on the creation of saliency maps-overlays to an input image that highlight regions most salient to the model in making its predictions. Among these, the most prominent are gradient-based methods (Simonyan et al., 2013; Sundararajan et al., 2017;6 Selvaraju et al., 2018) and masking based methods (Fong & Vedaldi, 2017; Dabkowski Gal, 2017; Fong & Vedaldi, 2018: Petsiuk et al., 2018: Chang et al., 2019; Zintgraf et al., 2017). In recent years, we have witnessed an explosion of research based on these two directions. With variety of approaches being proposed, framed and evaluated different ways, it has become difficult to assess and fairly evaluate their additive contributions.
In this work, we investigate the class of masking maasengg-based saliency methods, where we train a masking model to generate saliency maps based on explicit optimization objective. Using general formulation, we iteratively evaluate the extent to which recently proposed ideas in the literature improve performance. In addition to evaluating Supervised Object Localization (WSOL) metrics, the Saliency Metric (SM), and the more recently introduced Pixel Average Precision (PxAP; Choe al., 2020), we also test our final models against suite of "sanity checks" for saliency methods (Adebayo et al., 2018; Hooker et al., 2018).
models against the commonly used Weakly
Concretely, make four major contributions. (1) We find that incorporating both masked-in classification maximization and masked-out entropy maximization objectives leads to the best saliency maps, and continually training the classifier improves the quality of generated maps. (2) We find that the masking model requires only the top layers of the classifier to effectively generate saliency maps. (3) Our final model outperforms other masking- metrics. (4) We find that a small number of examples- as few as ten per class- is sufficient to train masker to within the ballpark of our best performing model.
methods on WSOL and PxAP

---

Under review as a conference paper at ICLR 2021
Training Macker
Imoge
class Fication socuracy
Maximine masked--in
Prediction
Nasker
Concatorate Cany Butrno
-3.5
Clocsiter
Clacs fferr
FD Softra
male
31.5
Mask
Classitter
16.
Spocs Thired
Reshiet Block
EExomple
Nesker
Rishist
Traihing Macker on excemp es per class
Mask
70.
Clasattee
Reckied Rlock
Maximize
Iritad Black
18:0
126
Classes
(A)
(C)
Figure 1: (A) Overview of the training setup for our final model. The masker is trained to maximize masked-in classification accuracy and masked-out prediction entropy. (B) Masker architecture. The masker takes as input the hidden activations of different layers of the ResNeet-50 and produces a mask of the same resolution as the input image. (C) Few-shot training of masker. Performance drops only slightly when trained on much fewer examples compared to the full training procedure.
2 RELATED WORK
Interpretability of machine learning models has been an ongoing topic of research (Ribeiro et al., 2016; Doshi-Velez & Kim, 2017; Samek et al., 2017; Lundberg et al., 2018). In this work, we focus on interpretability methods that involve generating saliency maps for image classification models. An overwhelming majority of the methods for generating saliency maps for image classifiers can be assigned two broad families: gradient- based methods and masking- baased methods.
Gradient-based methods, such as using backpropagated gradients (Simonyan et al., 2013), Guided Backprop (Springenberg et al., 2015), Integrated Gradients (Sundararajan al., 2017), GradCam (Sel- varaju et al., 2018), SmoothGrad (Smilkov et al., 2017) and many more, directly use the backpropa- gated gradients through the classifier to the input to generate saliency maps.
Masking based methods modify input images to alter the classifier behavior and use the regions of modifications as the saliency map. Within this class of methods, one line of work focuses on optimizing over the masks directly: Fong & Vedaldi (2017) optimize over perturbation mask for an image, Petsiuk et al. (2018) aggregates over randomly sampled masks, Fong & Vedaldi (2018) performs extensive search for masks of given size, while Chang et al. (2019) includes a counterfactual mask-infilling model to make the masking objective more challenging. The other line of work trains a separate masking model produce saliency maps: Dabkowski & Gal (2017) trains a model that optimizes similar objectives to Fong & Vedaldi (2017), Zolna et al. (2020) use continually trained pool of classifiers and adversarial masker to generate model-agnostic saliency maps, while Fan et al. (2017) identifies super-pin spperspieels from the image and then trains the masker similarly in an adversarial manner.
Salient Object Detection (Borji et al., 2014; Wang et al., 2019) is a related line of work that concerns identifying salient objects within an image as an end in itself, and not for the purpose of model interpretabilityy While it is not uncommon for these methods to incorporate pretrained image classification model to extract learned visual features, they often also incorporate techniques for improving the quality of saliency maps that are orthogonal to model interpretability. Salient object detection methods that are trained on only image-level labels bear the closest similarity to saliency map generation methods for model interpretability. Hsu al. (2017) and follow-up Hsu et al. (2019) train a masking model to confuse a binary imagecllasification model that predicts whether an image contains object or is 'background' image. Wang et al. (2017) apply a smooth pooling operation
2

---

Under review as a conference paper at ICLR 2021
and Foreground Inference Network (a masking model) while training an image classifier to generate saliency maps as a secondary output.
Evaluation of saliency maps The wave of saliency map research has also ignited research evaluation methods for these saliency maps as model explanations. Adebayo et al. (2018) and Hooker et. (2018) propose sanity checks and benchmarks for the saliency maps. Choe al. (2020) propose Pixel Average Precision (PxAP), a pixel-wise metric for scoring saliency maps that accounts for mask binarization thresholds, while Yang & Kim (2019) create set of metrics as well as artificial datasets interleaving foreground and background objects for evaluating the saliency maps. These works have shown that number of gradient-basee methods fail the sanity checks or perform no better than simple edge detectors. Hence, we choose to focus on masking-basse -based methods in this paper.
3 MASKING-BASED SALIENCY MAP METHODS
We start by building a general formulation of masking
wasking-based based saliency map methods. We take as given trained image classifier F - y. that maps from image inputs E = XX to class predictions ÿ E [0, 1] j"K evaluated against ground-truth y E (1 (1.... K). Our goal is to generate mask m E 00, 1] W for each image such that the masked-in image x C or the masked-out image r O 1m)m maximizes some objective based on output of a classifier given the modified image. For instance, could attempt mask out parts of the image to maximally deteriorate the classifier's performance. This mask m then serves as a saliency map for the image x. Concretely, the per-image objective can be expressed as:
arg min Aour Lou [F(x 3 (1 0)):pp).)) - Lin (F(x O 0F), + R(m), m
where Lout Lm the masked-out and masked-in in objectives over the classifier output, Aeut Am are hyperparameters controlling weighting of these two objectives, OF the classifier parameters, and R(m) a regularization term over the mask. The masked-in and masked-out losses, Lout and Lin- correspond to finding the smallest destroying region and smallest sufficient region as described in Dabkowski & Gal (2017). Candidates for Lout include negative classification cross-entropy and prediction entropy. For Lin the obvious candidate is the classification cross-entropy of the masked-in image. We set Am or Aus if we only have either a masked-in or masked-out objective.
The above formulation subsumes number of masking- based methods, such as Fong & Vedaldi (2017); Dabkowski & Gal (2017); Zolna al. (2020). We follow Dabkowski & Gal, amortize the optimization by training a neural network masker M x + m, and solve for:
arg min Aome Lom (F (Fxx (1- (m);;))) v) Lin (F 0M)); 0r), u) + (R;M 0m)), PM
where M is the masking model and Os its parameters. In our formulation, we do not provide the masker with the ground-truth label, which differs from certain other masking-based saliency works (Dabkowski & Gal, 2017; Chang et al., 2019; Fong & Vedaldi, 2018). In practice, we often desire model explanations without the availability of ground-truth information, we focus our investigation on methods that require only an image as input.
3.1 MASKER ARCHITECTURE
We use a similar architecture to Dabkowski & Gal and Zolna et al.. The masker takes as input activations across different layers of the classifier, meaning it has access the internal representation of the classifier for each image. Each layer of activations is fed through a convolutional layer and upsampled (with nearest neighbor interpolation) so they all share the same spatial resolution. All transformed layers are then concatenated and fed through another convolutional layer, upsampled, and put through sigmoid operation obtain mask of the same resolution as the input image. In all our experiments, we use a ResNet-50 (He et al., 2016) as our classifier, and the masker has access to the outputs of the five major ResNet blocks. Figure 1B shows the architecture of our models.
Following prior work (Fong & Vedaldi, 2017), we apply regularization on the generated masks to avoid trivial solutions such as masking the entire image. We apply LI regularization to limit the size of masks and Total Variation (TV) to encourage smoothness. Details can be found in Appendix A.
3

---

Under review as a conference paper at ICLR 2021
Model
OM
LE
SM
PXAP
Train- Validation Set
a) FIX
MaxEat
4..81.0.31
0.221:
b) FIX
MinClass (O)
62.54 4.29
54.8 5.15
0.251: 0.170
45.6+t 6.67
e) HIX
MaxClass ())
50.3+ 0.04
39.9 0.04
0.14+ 0.001
$5.4++0.10
d)
Max (O)
46.74 0.13
349-
0:2++
51.3+ 0.16
8) CA
MinClass (O)
45.94 0.06
34.1=0.05
0.17+ 0.004
54.8
CA
MaxClass
55.44 4.76
45.7= 5.30
0.30+
43.1+
8) FIX
MaxClass ()) MaxEnt (O)
46.74 0.02
38.7= 0.02
0.07+ 0.002
57.0+ 0.09
b) FIX
MaxClass MinCluss (O)
50.64: 1.70
40.2% 2,03
0.141 0.064
51.34 3.98
9) CA
MaxClass MaxFat (O)
45.04 0.06
33.4-000 0.06
0.12+ 0.044
60.2+
3) CA
MaxClass (0) + MinClass (O)
45.6d 0.09
34.1 0.10
0,17 0.004
60.6:t 0.14
18)
MaxClass
(O)
[
57.34: 0.07
0.40
() HIX
MaxClass (1) MaxEat (O)
+ Layer[5]]
53.6tt 0.03
43.8- 0.04
0.31+ 0.0011
44.0+t 0.10
FX
MaxClass MaxEnt (O)
Layer[[]
48.43: 0.03
37.8= 0.05
0.041 0.001
55.54 0.15
n) PIX
MaxClass (1) MaxEat (O)
+ Layer[4-5]
47.0+ 0.05
3600= 0.07
0.04+ 0.001
58.2+
0) CA
MaxClass (3) MaxdEnt (O)
+ Layer[]]
75.3 0.73
69.3 0.88
0.491 0.008
27.2 0.00
p)
MaxClass MaxEat
-Layer[3]
55.44
45.7=
0.27
g) CA
MaxClass MaxEat (O)
+ Layer[55
46.61 0.08
35 0.09
0.112 0.003
56.6:1 0.22
1) CA
MaxClass ()) MaxEan (O)
Layer[4-SS]
45.1 0.04
13.3 0.05
0.11 0.002
61.14 0.13
8) FIX
MaxClass MaxEat (O)
+ Layer[445]
Iol[
47.1 0.04
36 2= 0.04
0.08 0.002
58.7+ 0.07
t) FIX
MaxClass (1) MaxEEt (O)
+ Layerf4-51
IaGCC
49.74 0.05
39.2 0.06
0,112 0.002
52.9 0.15
0) FIX
MaxClass MaxEat (O)
Layer[4455]
+If[DFN]
49.94
39.5=
v) CA
MaxClass MaxEat (O)
+ Layerf44S]
IafiBlurl
46.01 0.12
34.5 0.13
0.07 0.004
59.511 0.19
M) CA
MaxClass (1) MaxEat (O)
+ Layer[44-5]
[nf[CGG]
45.4+ 0.05
33.6- 0.05
0.12+ 0.002
$6.7+ 0.08
x) CA
MaxClass MaxEnt (O)
Layerr44S]]
IIf[DFN]
49.0.1
0.021:
y) FIX
MaxClass (1) MaxEnt (O)
+ Layer|4-S1 51
Inf[CG
+ GS
52.8.1 0.24
43.0 0.30
0.19 0.003
40.6.1 0.20
2) MaxClass MaxEnt (O)
Layer[445]
[af[CAG]
48.94 0.28
38.1=
-0.01
19.3
20 Mask In Everything
50.9
50.9
0.44
27.2
Mask Nothing
100.0
100.0
4.72
c) Mask Center 50% Area
68.1
68.1
-0.09
36.7
Validation Set
D) & Vedaldi (2017)
R) Dabkoeski & Gal (2017)
36.9
0.32
P) Zolna et al. (2020)
48.6
36.1 1
G) Chang
(2019)
()) CAM
48.1
58.0
2) Our Best FIX
51.5
39.7
0.59
54,5
J) Our Best CA
48.4
35.8
0.52
59.4
Table 1:
Evaluation of masking-based saliency map methods. Each block captures one set of experiments. FIX indicates fixed classifier, CA (Classifie-AAnostti) indicates training against pool of continually trained classifiers. MaxClass (I), MinClass (O) and MaxEnt (O) are masked- in
confiiaationmmmiiizzattioon and masked-out entropy maximization objectives for the masker. Layer[-] indicates the layer or layers of classifier activations provided as input to the masker. Inf[-] indicates the infiller operation applied after masking- default otherwise is no infilling. Columns show mean and standard errors over 5 runs for evaluation metrics Official Metric (OM) and Localization Error (LE) for weakly supervised localization, Saliency Metric (SM) and Pixel Average Precision (PxAP). Underlined results are the best results within that block, while bold are the best results for data set, excluding baselines.
masked-out
3.2 CONTINUAL TRAINING OF THE CLASSIFIER
Because neural networks are susceptible to adversarial perturbations (Goodfellow et al., 2015), masking models can learn to perturb an input to maximize the above objectives for a given fixed classifier without producing intuitive saliency maps. While directly regularizing the masks is one potential remedy, Zolna et al. (2020) propose to train the masker against a diverse set of classifiers. In practice, they simulate this by continually training the classifier on masked images, retain pool of past model checkpoints, and sample from the pool when training the masker.
We adopt their approach and distinguish between masker trained against fixed classifier (FIX) and against a pool of continually trained classifiers (CA, for Classifier-Agnostie We highlight that saliency maps for FIX and CA address fundamentally different notions of saliency. Whereas FIX approach seeks a saliency map that explains what regions are most salient to a given classifier, CA approach tries to identify all possible salient regions for any hypothetical classifier (hence, classifier-agnosse agnostic).. In other words, a CA approach may be inadequate for interpreting a specific classifier and is better suited for identifying salient regions for a class of image classification models.
4

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTAL SETUP
We perform our experiments on the official ImageNet training and validation set (Deng et al., 2009) and use bounding boxes from the ILSVRC' 14 localization task. Because we perform a large number of experiments with hyperparameter search to evaluate different model components, we construct a separate held-out validation set of 50,000 examples (50 per class) from the training set with bounding box data that we use as validation for the majority of our experiments (which we refer to as our "Train- Validation" set) and use the remainder of the training set for training. For each model configuration, we train the models 5 times on different random seeds and report the mean and standard error of the results, We reserve the official validation set for the final evaluation.
4.1 EVALUATION METRICS
Weakly-supervised object localization task metries (WSOL) is a common task for evaluating saliency maps. It involves generating bounding boxes for salient objects in images and scoring them against the ground-truff -ruth bounding boxes. To generate bounding boxes from our saliency maps, we binarize the saliency map based on the average mask pixel value and use the tightest bounding box around the largest connected component of our binarized saliency map. We follow the evaluation protocol in ILSVRC 14 computing the official metric (OM), localization error (LE) and pixel-wise Fl score between the predicted and ground-truth bounding boxes.
Saliency metric (SM) proposed by Dabkowski & Gal (2017) consists of generating bounding box from the saliency map, upsampling the region of the image within the bounding box and then evaluating the classifier accuracy the upsampled salient region. The metric is defined as s(a, p) = (2mmm(x(a 0.05)) log(p), where a is the size of the bounding box, and p is the probability the classifier assigns the true class. This metric can be seen measuring masked-in and upsampled classification accuracy with a penalty for the mask size. We use the same bounding boxes as described in WSOL for consistency.
Pixel Average Precision (PxAP) proposed by Choe et al. (2020) scores the pixel-wise masks against the ground-truth masks and computes the area under the precision-recall curve. This metric is computed over mask pixels rather than bounding boxes and removes the need to threshold and binarize the mask pixels. PxAP computed over the OpenImages dataset (Benenson et al., 2019) rather than ImageNet because it requires pixel-level ground-trun truth masks.
5 EVALUATION OF SALIENCY MAP METHODS
To determine what factors and methods contribute to improved saliency maps, we perform series of evaluation experiments in a cascading fashion. We isolate and vary one design choice at a time, and use the optimal configuration from one set of experiments in all subsequent experiments. Our baseline models consist of a masker trained with either a fixed classifier (FIX) or a pool of continually trained classifiers (CA). As WSOL is the most common task for evaluating saliency maps, we use LE as the metric for determining the 'best' model for model selection. We show our model scores across experiments in Table 3. Each horizon block represents a set of experiments varying one design choice. The top half of the table is evaluated on our Train- valinVaaiiation split, while the bottom half is evaluated on the validation data from ILSVRC '14. 14.
Masking Objectives (Rows a-f) We first consider varying the masker's 's training objective, using only one objective at time. We use the three candidate objectives described in Section 3: maximizing masked-in in accuracy, minimizing masked-out out accuracy and maximizing masked-out entropy. For masked-out objective, we set Aour 1, Aln O, and the opposite for masked-in in objectives. For each configuration, we perform random hyperparameter search over the L11 mask regularization coefficients AM and Àrv as well as the learning rate and report results from the best configuration from the Train-Validadee Validation set. More details on hyperparameter choices can be found in Table 2.
Consistent with Zolna et al. (2020), we find that training the classifier along with the masker improves the masker, with CA models generally outperforming FIX models, particularly for the WSOL cessifiiationmmximmizattoo
metrics. However, the classification-mann maximization FIX model still performs comparably with its CA counterpart and in fact performs best overall when measured by SM given the similarity between the training objective and the second term of the SM metric. Among the CA models, entropy- maximization and tassificatoonmmniiizztii perform the best, while the classification maxiaiiiiiiiniiiiiiiiiiooo
5

---

Under review as a conference paper at ICLR 2021
objective performs worst. On the other hand, both mask-out objectives perform extremely poorly for a fixed classifier. We show how different masking objectives affect saliency map generation in Figure 2.
PIX
PIX
FIX
CA
Mit
MAREIL (OI
MESCIAS OO
MELESE (O)
selamanzzsr
Imput
Appenudia
Mask Piael Distribution 635
Figure 2: Varying the masking objective for a fixed classifier (FIX) and blassfierr-gnosstc (CA) settings. Top: Examples of masked-in inputs and masks. The CA models generally produce more contiguous masks, and combining both a masked-in and masked-ou objective works best based on quantitative evaluation. Bottom: Distribution of mask pixel values across Train- Train-Validatio Validation set. We quantize the values into 10 buckets in the bar plot in blue and also show the empirical CDF in red. Most models produce highly bimodal distributions, with most pixel values close
1.
Combining Masking Objectives (Rows g-j) Next, we combine both masked-in and masked-out objective during training, setting Aou Am 0.5. Among the dual-objective models, entropy- maximization still outperforms lassification-minniizzii both masked-in for both FIX and CA models, consistent with Dabkowski & Gal (2017). From our hyperparameter search, we also find that separately tuning Mn and XM ous is highly beneficial (see Table 2). We use the classification canattiinmmaiiiizattin and citoop--mxxmmzzttion dual objectives for both FIX (Row g) and CA (Row i) models in subsequent experiments.
as masked-out objective. Combining and masked-out entropy-maximization performs best
Varying Observed Classifier Layers (Rows k-r) We now vary which hidden layers of the clas- sifier the masker has access to. As described in Section 3.1, the masking model has access to hidden activations from five different layers of a ResNet-50 classifier. To identify the contribution of train completely new masking models with information from each layer to the masking model, access to only a subset of the classifier layers. We number the layers from 1 to 5, with being the earliest layer with the highest resolution (56 x 56) and 5 being the latest (7 7). We show relevant subset of the results from varying the observed classifier layers in Table 3. The full results can be found in the Table and show examples of the generated masks in Figure 3.
Masking models with access to activations of later layers starkly outperform those using activations from earlier layers. Whereas the Layer[3], Layer[4] and Layer[5] models are still effective, the Layer[1] and Layer[2] models tend to perform poorly. Similarly. we find that the best cascading combination of layers is layers 4 and CA models, and 3-5 for FIX models (Rows n, x), slightly but consistently outperforming the above models with all layers available to the masker. This suggests that most of the information relevant for generating saliency maps is likely contained within the later layers. For simplicity, we use only classifier layers 4 and 5 for subsequent experiments.
Counterfactual Infilling and Binary Masks (Rows s-z) Chang et al. (2019) proposed generating saliency maps by learning a Bernoulli distribution per masking pixel and additionally incorporating a counterfactual infiller. Agarwal& & Nguyen (2019) similarly uses an infiller when producing saliency maps. First, we consider applying counterfactual infilling to the masked images before feeding them to the classifier. The modified inputs are Infill(X (1 m), (1 m)) and Infill(X m, m) for

---

Under review as a conference paper at ICLR 2021
Input
Layer=1
Layer-
Layer=
Layer=4
Layer=
All Layers
Tench
Input
Radilator
Figure 3: Examples of varying the layers of classifier activations that the masker takes as input, from Layers I (closest to input) to 5 (closest to output). Each column is from a separately trained model. Using activations from higher layers leads to better saliency maps, despite having lower resolution. All Layers has access to all 5 layers, but does not perform better than simply using layers 4 and 5.
masked-out and masked-in infilling respectively, where Infill is the infilling function that takes as input the masked input as well as the mask. We consider three infillers: the Contextual Attention GAN (Yu et al., 2018) as used in Chang et al. DFNet (Hong et al., 2019), and Gaussian blur infiller as used in Fong & Vedaldi (2018). Both neural infillers are pretrained and frozen.
For each infilling model, we also train a model variant that outputs a discrete binary mask by means of Gumbel-Softmax layer (Jang et al., 2017: Maddison et al., 2017). We experiment with both soft and hard (Straight-Throughh Gumbel estimators, and temperatures of {0.01, 0.05, 0.1, 0.5}.
We show a relevant subset of the results in the fourth and fifth blocks of Table 3, and examples in Figure 6 and Figure 7. We do not find major improvements from incorporating infilling or discrete masking based on WSOL metries, although we do find improvements from using the DFN infiller for SM. Particularly for CA, because the classifier is continually trained to classify masked images, it is able to learn to both classify unnaturally masked images as well as to perform classification based on masked-out evidence. As result, the benefits of incorporating the infiller may be diminished.
5.1 EVALUATION ON VALIDATION SET
Based on the above, we identify simple recipe for good saliency map generation model: (1) use both masked-in classification maximization and masked-out entropy maximization objectives, (2) use only the later layers of the classifier as input to the masker, and (3) continually train the classifier. To validate the effectiveness of this simple setup, we train a new pair of FIX and CA models based on this configuration compare the results to other models in the literature in the bottom block of Table 3. Consistent with above, the CA model outperforms the FIX model. also outperforms other saliency map extraction methods on WSOL metrics and PxAP. We highlight that some models we compare to (Rows E, G) are provided with the ground-truth target class, whereas our models are not-this may explain the underperformance on certain metrics such as SM, which is partially based on classification accuracy.
the full training set and evaluate on the actual ILSVRC '44 validation set. We
5.2 SANITY CHECKS
Adebayo et al. (2018) and Hooker et al. (2018) propose "sanity checks" to verify whether saliency maps actually reflect what information classifiers use make predictions and show that many proposed interpretability methods fail these simple tests. We apply these tests to our saliency map models to verify their efficacy. On the left of Figure 4, we show the test proposed by Hooker et al., where we remove the top 1% of pixels from training images based on our generated saliency maps and use them to train entirely new classifiers. If our saliency maps truly identify salient portions of the image, increases. Both FIX and CA methods pass this test, with classifier accuracy falling precipitously as we mask out more pixels. On the right of Figure 4, we perform the the Model Parameter Randomization Test (MPRT) proposed by Adebayo et al.. We randomize parameters of successive layers of the
(ROaR)
should see large drops in classifier performance as t
'The publicly released CA-GAN is only trained rectangular masks, but Chang et al. nevertheless found positive results from applying it, so we follow their practice. DFNet is trained irregularly shaped masks.
7

---

Under review as a conference paper at ICLR 2021
classifier, starting from upper-most logits layer to the lowest convolutional layers, and generate saliency maps using the partially randomized classifiers. We then compute the similarity of the saliency maps generated from using the partially randomized classifier, and those using the original classifier. Our saliency maps become less similar as more layers are randomized, passing the test. The results for the Data Randomization Test (DRT) can be found in Table 5.
Remova-Andd-Rtrri
MPRT
1.0
1.0
FIX. Top-1
0.5
Correl [ABS]
FIX. Tap-5
Accuracy
GA. Top
0.4
Top
-0.5
100
MPRT
SSIM
Similanily
HDGs
0.2
-0.5
20
40 60
80
original
Block 4
Block-3
Block-3
Block1
Cow 1
% of input features replaced
Figure 4: Our saliency maps pass recently proposed sanity checks. Left: RemOve-Ane Aave-And-Retraill evaluation from Hooker et al. (2018). We train and evaluate new ResNet-50 models based on training images with the top 1% of most salient pixels removed. Removing the most salient pixels hurts the new classifier's performance, and the CA model is more effective at removing salient information. Right: Model Parameter Randomization Test (MPRT) from Adebayo et al. (2018). We randomize parameters of successive layers of the classifier, and compute the similarity between the saliency maps generated using the original and modified classifier. Similarity falls as more layers are randomized.
6 FEW-SHOT EXPERIMENTS
Given the relative simplicity of our best-performing saliency map models and the fact that the masker uses only the top layers of activations from the classifier, hypothesize that learning to generate saliency maps given strong classifier is a relatively simple process.
To test this hypothesis, and unique examples that the masker is trained on. The ImageNet dataset consists of 1,000 object classes, with up 1,250 examples per class. We run set of experiments restricting both the number of unique classes seen as well as the number of examples seen per class while training the masker. We also limit the number of training steps be equivalent to one epoch through the full training set. Given the randomness associated with subsampling the examples, we randomly subsample classes and/or examples times for each configuration and compute the median score over the 5 runs. We report results on the actual validation set for ILSVRC 14 (LE) and test set for OpenImages (PxAP). Examples of saliency maps for these models can be found in Figure
a of experiments severely limiting the number of training steps
We show the results for our CA model in Figure and for the FIX model in Figure 8. Strikingly, we find that very few examples are actually required to train a working saliency map model. In particular, training on just 10 examples per class produces model that gets only 0.7 LE more than using all of the training data and only 2.7 more than the fully trained model.
On the other hand, we find that the diversity of examples across classes is bigger contributor to performance than the number of examples per class. For instance, training on 10 examples across all 1,000 classes gets an I.E of 38.5, which is lower than training on 125 examples across only 100 classes. A similar pattern can be observed in the PxAP results.
Above all, these few-shot results indicate that training effective saliency map model can be significantly simpler and more economical than previously thought. Saliency methods that require training separate model such Dabkowski & Gal (2017) and Zolna al. (2020) are cheap to run inference, but require an expensive training procedure, compared to gradient-based saliency methods or methods involving a per-example optimization. However, training a masking model can be lightweight procedure as we have demonstrated, then using masking models to generate saliency maps can now be a cheap and effective model interpretability technique.
8

---

Under review as a conference paper at ICLR 2021
Localization Error (LE)
Pixel Average Precision (PxAP) f
58.5
54.0
57.0
66.2
10
29.8
35.5
37.4
41.7
# Clackes
100
53.7
46.1
40.3
40.4
100
33.0
43.7
54.5
57.0
1000
44.3
38.5
37.3
37.8
1000
45.7
54.3
58.8
57.9
10
125
1250
10
125
1250
Examples
Examples
Figure 5: Training the saliency map models (CA) using fewer examples per class and/or fewer classes out of the 1000 in the training set. Results reported are the median over 5 runs with different samples of examples. Lower LE and higher PxAP are better. Models can be trained generate good saliency maps with surprisingly few number of training examples. For comparison, the LE and PxAP for the fully trained CA model are 35.8 and 59.4 respectively.
DISCUSSION AND CONCLUSIONS
In this work, systematically evaluated the additive contribution of many proposed improvements
to masking- based saliency map methods. Among the methods we tested, we identified that only the following factors meaningfully contributed improved saliency map generation: (1) using both masked-in and masked-out objectives, (2) using the later layers of the classifier as input and (3) continually training the classifier. This simple setup outperforms other methods on WSOL metrics and PxAP, and passes suite of saliency map sanity checks.
Strikingly, we also found that very few examples are actually required to train a saliency map model, and training with just 10 examples per class can achieve close to our best performing model. In addition, our masker model architecture is extremely simple: a two-layer ConyNet. Together, this suggests that learning masking- maseeng-based saliency map extraction may be simpler than expected when given access to the internal representations a classifier. This unexpected observation should make us reconsider both the methods for extracting saliency maps and the metrics used to evaluate them.
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In NeurlPS, 2018.
Chirag Agarwal and Anh Nguyen. Explaining image classifier's decisions using generative models. arXiv e-prints, art. arXiv:1910.044 1910.0456,, October 2019.
Rodrigo Benenson, Stefan Popov, and Vittorio Ferrari. Large-scale interactive object segmentation with human annotators. In CVPR, 2019.
Ali Borji, Ming-Ming Cheng, Huaizu Jiang, and Jia Li. Salient object detection: survey. CoRR, abs/1411.588 2014. URL htt.http//arxtie http://wxww..ooogb//////ao 1111a111./1111111111111/11111111 5878.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image classifiers by counterfactual generation. In /CLR, 2019.
Junsuk Choe, Seong Joon Oh, Seungho Lee, Sanghyuk Chun, Zeynep Akata, and Hyunjung Shim. Evaluating weakly supervised object localization methods right. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020. to appear.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In NIPS, 2017.
J. Deng, W. Dong, R. Socher, L.-. Li, Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009.
Finale Doshi-Velez and Been Kim. Towards rigorous science of interpretable machine learning, 2017.
Lijie Fan, Shengjia Zhao, and Stefano Ermon. Adversarial localization network. 2017. URL http://1 http://ijfffaaa/e me/aammmmeeffee wwt.emeeeeeeettnnnnnneee ps L-LLD. .d

---

Under review as a conference paper at ICLR 2021
Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In CVPR, 2018.
Ruth Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In ICCV, 2017.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Xin Hong, Pengfei Xiong, Renhe Ji, and Haoqiang Fan. Deep fusion network for image completion. In ACM MM, 2019.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. Evaluating feature importance estimates, arXiv: 1806.10758, 2018.
K. Hsu, Y. Y. Lin, and Y. Chuang. Weakly supervised salient object detection by learning classifier-drig -riive map generator. IEEE Transactions on Image Processing, 28(1 28441))533--4 2019. doi: 10.1 19.110//TP.2019.99111
Kuang-Jui Hsu. Yen- Yu Lin, and Yung- Yu Chuang. Weakly supervised saliency detection with A category-driven map generator. In British Muchine Vision Conference 2017, BMVC 2017, London, UK, September 4-7, 2017, BMVA Press, 2017. URL wwtpt:tppp/..o. http:s://www.ddropbre com/
s/ //6bbdd644711 (11/444711pp///22 02700111100 pdfppd==
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In ICLR, 2017.
Alex Krizhevsky, Ily Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In NIPS, 2012.
Scott M. Lundberg, Gabriel G. Erion, and Su-In Lee. Consistent individualized feature attribution for tree ensembles. arXiv: 1802.03888, 2018.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. In ICLR, 2017.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. In BMVC, 2018.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In KDD, 2016.
Wojeiech Samek, Thomas Wiegand, and Klaus-Robert Müller. Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. CoRR, tbs//708.00296, 2017. URL
org/ abs/ 10001170000 08299 96
Ramprasaath Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? visual explanations from deep networks via gradient- hased localization. In ICCV, 2018.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv: 1312.6034, 2013.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. CoRR, abs/1706.03885, 2017. URL httu:://axxii..or abs/1706 03825.
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox. and Martin Riedmiller. Striving for simplicity: The all convolutional net. In ICLR, 2015.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, 2017.
10

---

Under review as a conference paper at ICLR 2021
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
L. Wang, H. Lu, Y. Wang, M. Feng, D. Wang, B. Yin, and X. Ruan. Learning to detect salient objects with image-level supervision. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3796-3805, Los Alamitos, CA, USA, jul 2017. IEEE Computer Society, doi: 10.1 10.1109/C2.20 1109/ 21199//VR.222 2017.
URL
Wenguan Wang, Qiuxia Lai, Huazhu Fu, Jianbing Shen, and Haibin Ling. Salient object detection in the deep learning era: An in-depth survey. CoRR, abs/1904.09 146, 2019. URL http: //aaxxiv.rrg//baaae
9044..... 09146.
Mengjiao Yang and Been Kim. BIM: towards quantitative evaluation of interpretability methods with ground truth. arXiv: 1907.09701, 2019.
Jiahui Yu, Zhe Lin, Jimei Yang. Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In CVPR, 2018.
Luisa M. Zintgraf, Taco Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. In ICLR, 2017.
Konrad ZoIna, Krzysztof Geras, and Kyunghyun Cho. Classifier-agnostic saliency map ex- traction. Computer Vision and Image Understanding, 196:102969, 2020. ISSN 1077-3142. doi: http:///i..og//...000j...v... 102969. URL http: www. encedi rect. com/ sci artene//errcdeeess////0133322220

---

Under review as a conference paper at ICLR 2021
A MODEL DETAILS
A.1 MASK REGULARIZATION
Without regularization, the masking model may learn simply mask in or mask out the entire image, depending on the masking objective. We consider two forms of regularization. The first is L1 regularization over the mask pixels m, which directly encourages the masks to be small in aggregate. The second is Total Variation (TV) over the mask, which encourages smoothness:
TV(m) Σ (ms. (mms ma,+1)2 + 1.j
> (may - -m;11))² E
where j are pixel indices. TV regularization was found to be crucial by Fong & Vedaldi (2017) and Dabkowski & Gal (2017) to avoid adversarial artifacts. Hence, we have:
R(m) Aa ||m|| ArvTV(m).
(1)
Following Zolna al. (2020), only apply L1 mask regularization if using masked-in objective and the masked-in image is correctly classified, or we have a masked-out objective and the masked-out image incorrectly clasifiid-otheeriise, no LI regularization is applied for that example. In cases where we have both masked-in and masked-out objective, we have separate AM and Mou regularization coefficients.
A.2 CONTINUAL TRAINING OF THE CLASSIFIER
We largely follow the setup for training classifier-agnostic (CA) models from Zolna et al. (2020). Notably, when training on the masker objectives, we update f but note fr, to prevent the classifier from being optimized the masker's objective. We maintain pool of 30 different classifier weights in our classifier pool. We point the reader to Zolna al. (2020) for more details.
A.3 HYPERPARAMETERS
We show in Table 2 the space of hyperparameter search and hyperparameters for the best results, as show in the Table in the main paper. We performed random search over XM AM,in- and \tV.
Aside from the hyperparameters shown in Table 2, we used a lcarning rate of 0.001 and a batch size of 72. We trained for 3.4 epochs (17 epochs on 20% of data) for all Train- valin-Validttion experiments and for 12 epochs for the Validation set experiments. Likewise, we use learning rate decay of 5 for Train- Val experiments and 20 for Validation set experiments. For dual objective models, used Agut Xin 0.5. We use the Adam optimize with 0.9 momentum and le-4 weight decay.
12

---

Under review as a conference paper at ICLR 2021
Hyperparamdters
Max (O)
Astza
[1.4.10. 15.00.5.000 Acv
10.0.0001, 0.001,,,..00000.0000000 0.01,
D) FIX
MinClass (O)
Aotza
[1.5,10 15. 30) Avv
00,0.000,,
0.005,,, 0.09
8) FIX
MaxCless (1)
[1.5.00 15.
Xrv
00.0.0001 0.001, 0.005, 0.000,00000.00 0.00
MaxEnt (O)
Astas
(1.5.10
0.00.001,
0.000.00000.0011......
CA
MinClass
Ausza
11.5, 15. 30) Acv
0,00000...000..00000 0.05)
CA
MaxClnss (1)
Auti
(1,4,10, 15, 30) Agv
10,0.0001,
FIX
MaxClass MaxEat (0)
(1.5, 15), = 5010, 15),
(0,0,0001, 0.006,,
b)) FIX
MaxClass (l) MinClass (O)
Astia
(1,5, 10, 13). Asta [1,5, 15).
Anv
(0.0.0001.0.0000 0,000,00.000,,000000 0.000.000...01.1.000 0.05)
1A
MaxClas MaxEat (0)
(1,5, A3) = (1,5,
15),
(0.0.0001. 0.001, 0.005...0005....11 0.000..0005
1)
MaxClass (0)
(O)
Astua (1,5, 10, 12). Assa [1.5, 15].
Arv (0,0.000), 0.00011 0.005, 0.01,.0000005
R)FXX
MaxClass Maxiar (0) Layed[]
10.
1. 0.001
Maxiat (0)+ Layer33
Auize Aota
1.Arr
m) FIX
MaxCCass (0) MaxEat (0) La
Astan 10, 10. 1.Arv 0.000
6) FIX
MaxClass (1 + MaxEar (0) Layer[4-5]
Antas 10. Jotia
1.Arr 0.001
MaxClass MadEat (0) Layerf]
-0.001
9)
MaxClass
+0y+Laye1
Auize A5,a 11,Arv
CA
MaxClass (1) Maximn (O) Layer[5]
Astae: 15. 15, 1.,AT 0.001
2) CA
MaxClass (0+ + MaxEat (0) Lay
Autze 15. Asta
11,Arv 0.001
FIX
MaxCless MaxPar Lay= Blor]
t)
MaxClass
(O) ---5
Jutar Auca
11,ArV
4) FIX
MaxClass (I) Maxiat + 4-51 In[[]N]
Jatae 10, Jocaa
1,Ary 0.001
v) CA
MaxClass (I) MaxFar (0) Layert(44) [nf[Blur]
Jate 15, Aara
1,A 00.001
CA
MaxClass MaxEat (0) Layerf[444 Inf[CA]]
15.
CA
MaxClass
(O) Layer(445) Inf[DNN
Autzs 15,
1.Ary
y) FIX
MaxClas (1) Max (0) 100+++yye443)) IINCCA) + GS
AMae = 10, Asta Ar = 0.001, (0.01, 0.0.0...550011000,,. 08 (4,h)
MaxClass MaxEat (0) Layer[1-5] [-]] In[CCG) GS
15, Anta 1. 0..00. (0.01.0.05.0.110...). GS (g.b)
Valudation Set
Autue
1,
B) Our Best CA
Potax 15, Aoca 1,Ary 0.001
Table 2: Hyperparameter search space and chosen hyperparameters. Sets of values in a row indicate grid search over all combinations in that row. Where there is a search, the hyperparameters for the best model, corresponding to results shown in Table are underlined.
B SUPPLEMENTARY RESULTS
B.1
B.2 VARYING OBSERVED CLASSIFIER LAYERS
We show the full set of per-layer and layer-combination results in Table 4.
B.3 COUNTERFACTUAL INFILLING AND BINARY MASKS
We show examples of saliency maps generated with counterfactual infillers in Figure 6 and incorpora- tion of Gumbel-Softman to generate masks with binary pixel values in Figure 7.
B.4 SANITY CHECKS
We show in Table 5 the results for the Data Randomization Test (DRT) proposed by Adebayo et al. (2018). Here, we train a new classifier the same training data but with labels shuffled across all images. We find that the similarity of saliency maps generated given regularly trained classifier compared to those given classifier trained on shuffled labels is low, indicating that the saliency maps reflect information learned from a well-formed image classification task.
B.5 FEW-SHOT EXPERIMENTS
We show in Figure 8 the results for few-shot experiments using the FIX model configuration. We similarly find that very few examples are needed to train good saliency map model.
13

---

Under review as a conference paper at ICLR 2021
Model
F1
Avg Mask
Train- Validation Set
a) FIX
MaxEnt (O)
60.6 0.1
56.3+ 1.7
b) FIX
MinClass (0)
34.2 10.2
31.1 16.5
C) FIX
MaxClass (I)
21.6 0.1
11.2 0.1
d) CA
MaxEnt (O)
63.1 0.0
51.1 0.2
e) CA
MinClass (O)
64.1 0.1
50.7+ 0.3
CA
MaxClass (I)
49.2 11.9
48.61 11.9
9) FIX
MaxClass (I) + MaxEEn (0)
53.64
35.2
FIX
MaxClass (I) + MinClass (0)
30.7+ 7.2
27.5+
1)
MaxClass (I) + MaxEnt (O)
62.91 0.1
46.6 0.3
MaxClass (I) MinClass (0)
64.8 0.1
52.9 0.3
FIX
MaxClass (I) MaxFint (0)
[Iaeer]]]
$7.7+ 0.1
85.5+ 1.3
1)
FIX
MaxClass (I) MaxFnt (0)
+Layer]3]
53.4
51.3
FIX
MaxClass ()) + MaxEnt (0)
+ -Layer[5]
$7.9
40,9
FIX
MaxClass (I) + MaxEnt (O)
+ +Layer[455]
5.11
38.0=
MaxClass (I) + MaxEnt (O)
+ [Layer[]]
1.34
0.8+
P) CA
MaxClass (I) + MaxEnt (O)
+ +Layer[3]
55.9+
54.9
9) CA
MaxClass (I) + MaxEnt (0)
+ +Layer[5]
63.1 0.1
48.61
CA
MaxClass (I) MaxEEn (0)
+ Laaer[4455]
63.02 0.1
46.31 0.0
F)F
FIX
MaxClass (I) + MaxEnt (0)
+ Layer[4-5]
Inf]Blur]
54.5
36.6
F)X
MaxClass (I) + MaxEnt (0)
+ 5]
+ [n[[CA]]
0.1
42.4
b) FIX
MaxClass (I) + MaxEnt (0)
+ 5a
+ [n[[DNN]
52.11
32.5
v) CA
MaxClass (I) + MaxEnt (O)
+ +aaerr44551
+ Bnt]Blur]
60.7+
42.1+
W) CA
MaxClass (I) + MaxEnt (O)
+ +1ayer44-51
+ Inf[CAG]
61.9+ 0.1
46.7+
x) CA
MaxClass (I) + MaxEnt (0)
+ Laaerr44-51
[nf]DFN]
53.2 0.2
32.7 0.2
FIX
MaxClass (l) MaxEnt (0)
Layer[4-5]
[n[[CG]
GS
59.71 0.2
$7.2
2) CA
MaxClass (l) MaxEnt (0) Layer[4-5]
In[[CG]]
GS
57.7+
39.1 0.2
Table 3:
Evaluation of masking based saliency map methods. Each block captures set of experiments. FIX indicates fixed classifier, CA (Classifier-Agnostcc) indicates training against pool of continually trained classifiers. MaxClass (I), MinClass (O) and MaxEnt (O) masked- in clasiffiation-maxinne maximizaiionn masked-out glassfiiattionmmiiiiiii
and masked-out entropy maximization objectives for the masker. Layer[-) indicates the layer layers of classifier activations provided as input to the masker. Inf[-] indicates the infiller operation applied after masking- default otherwise no infilling. Columns show average mask magnitude. Underlined results are the best results within that block, while bold are the best results for data set, excluding baselines.
and standard errors over runs for Fl and the
Blur
CA-GAN
DFNet
Mosked Out Masked Out
Massod in.
Mask
Masked Ou: Masked Out. Vacked in
Nisk
Masked Out
Nasked out.
Nasked in
Kask
Input
FX
Tiger
Input
Figure 6: Saliency maps using various infilling methods for counterfactual generation. Following Chang et al. (2019); Agarwal & Nguyen (2019), we infill masked out portions of the image and provide the resulting infilled image to the classifier. Infillers are hypothesized to help training by making the classifier inputs look closer to natural images as well as forcing the masker mask out all evidence of a salient object. FIX indicates a fixed classifier, CA (Classifier- Aggiisrrc indicates training against a pool of continually trained classifiers. We not find quantitative improvements from incorporating an infiller.
14

---

Under review as a conference paper at ICLR 2021
Model
OM
LE
FI
SM
PXAP
Mask
Train- Vrain-Validation Set
FIX
MaxClass (I) MaxFnt (O)
[I]yer[I]
57.3
48.2
57.6
0.40
34.94
84.0
FIX
MaxClass (1) MaxEnt (O)
LLayer[2]
55.8
53.5
0.37
38.26
61.5
FIX
MaxClass (1) MaxEnt (O)
Laaer[3]]
53.6
43.9
53.5
0.31
43.97
51.6
d FIX
MaxClass (1) MaxEnt (O)
LLyer[44]
47.9
36.9
55.2
0.15
58.83
41.2
e) FIX
MaxClass (1) MaxEnt (O)
LLyer[5]
48.5
37.9
57.9
0.04
55.53
40,9
FIX
MaxClass (1) MaxEnt (O)
Layer[2-S]
46.7
35.8
53.6
0.08
56.75
35.5
FIX
MaxClass (1) MaxEnt (0)
La
46.7
35.7
54.6
0.06
57.53
36,2
5) FIX
MaxClass (1) MaxEnt (O)
Laycr[4-S]
47.0
36.1
56.0
0.04
58.11
37.8
CA
MaxClass (1) MaxEnt (O)
Layer[I]
77.5
71.9
1.3
0.49
27.24
0.8
3) CA
MaxClass (1) MaxEnt (0)
Layer[2]
71.9
65.1
1.3
0.62
27.25
0.8
X) CA
MaxClass (1) MaxEnt (0)
Layer[3]
54.9
45.1
56.2
0.27
44.51
54,9
CACA
MaxClass (1) MaxEnt (O)
Layer[4]
46.7
35.1
60.8
0.16
57.94
48.2
m) CA
MaxClass (1) MaxEnt (O)
Layer[5]
46.4
35.2
63.2
0.11
56.95
48.5
2) CA
MaxClass (1) MaxEnt (O) + Layer[2-5]
45.4
33.6
63.1
0.13
60.88
47.2
o) CA
MaxClass (1) MaxEnt (O) Layer[3-5]
45.3
33,4
63.3
0.12
61.47
47.1
CA
MaxClass (1) MaxEnt (O) Layer[4-5]
45.0
33.2
63.1
0.11
61.24
46.4
Table 4:
Evaluation of masking masking-based saliency map methods, varying the layers provided to the masker. FIX indicates fixed classifier, CA (Classifie-Agnostte) indicates training against pool of continually trained classifiers. MaxClass (I), MinClass (O) and MaxEnt (O) are masked- in
and masked-out entropy maximization objectives for the masker. Layer[-] indicates the layer or layers of classifier activations provided as input to the masker. Inf[-] indicates the infiller operation applied after masking-the default otherwise is no infilling. Columns show evaluation metrics Official Metric (OM), Localization Error (LE) and F1 for weakly supervised localization, Saliency Metric (SM) and Pixel Average Precision (PxAP). Mask indicates numerical average of masking pixel values. Underlined results indicates the best results within that block, while bold indicates best results data set, excluding baselines.
masked-out
Input
Infile
Inilert Cb
Input
FIX
Figure 7: We apply the Gumbel--Softaax trick to train the masker to produce binary masks, infilling using the CA-GAN. As seen in Figure 2, most models in experiments produce mask pixel values of 0 or I, so the benefits of explicitly learning a discrete output distribution are limited.
Model
Rank Correl(Abs)
Rank Correl(No Abs)
HOGS Similarity
SSIM
FIX
-0.069
0.037
0.488
0.001
CA
0.129
0.133
0.519
0.022
Table 5: Data Randomization Test. The saliency map methods are applied to both a regular classifier, as well classifier trained on randomly shuffled labels, and the similarity of the generated saliency maps are measured. Both FIX and CA methods show low similarity between the saliency maps generated by regular and shuffled classifiers.
15

---

Under review as a conference paper at ICLR 2021
Localization Error (LE)
Pixel Average Precision (PxAP) f
64.7
51.4
48.2
78.9
29.6
40.3
43.5
40.7
a Claisens
51.6
44.8
42.8
43.5
100
37.2
45.0
52.9
58.5
1000
44.6
40.3
38.3
38.7
1000
45.1
52.2
56.5
55.9
10
125
1250
10
125
1250
Examples
Examples
Figure 8: Training the saliency map models (FIX) using fewer examples per class and/or fewer classes out of the 1000 in the training set. Results reported are the median over 5 runs with different samples of examples. Lower LE and higher PxAP are better. Models can be trained generate good saliency maps with surprisingly few number of training examples.
1000 classes examples
100 classes 10 examples
1000 classes
100 classes
1000 classes
100 classes
1000 dasses
Full
Input
examples
125 examples
125 examples
1250 examples
1250 examples
Training
Input
FIX
Meu Longe
FIX
RVC feice
Input
FIX
Figure 9: Examples of saliency maps computing from models trained with less data. Columns correspond to the models shown in Figure 5 and Figure 8. Even models trained with very few examples per class produce saliency maps similar to the fully trained model.
16


Reviews:
[{"review_id": "tfVaT9VxvyX", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "-Summary: \nThis paper investigates the previous masking-based saliency map methods. By detailly formulate the masking-based saliency methods and sufficient experimentation, the paper gives a simple formulation and practical training strategies.\n\n-Strength: \nThe paper is well organized and the presentation is easy to follow.\nThe experiments are comprehensive.\n\n-Weakness:\nAlthough the experiments are enough, the paper simply adopts the previous masking-based methods, evaluation metrics, architectures, and sanity check analysis. The theoretical analysis is insufficient.  More theoretical experiments instead of performance analysis need to be added.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "qYzAHWLf7C-", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an approach to improve masked based prediction explanation. They train an auxiliary model which predicts a mask that must satisfy two terms 1) maximize classification accuracy when applied to the image and 2) maximize entropy over softmax when the inverse mask is applied to the image (they also experiment with minimizing the classification accuracy instead here).\n\nThe paper also positions their work to be classifier agnostic in the text which is not clear to me. I think this aspect is a negative because it is not in the list of contributions and seems like a distraction of a remnant of an old direction of the paper:\n\n> \"Whereas a FIX approach seeks a saliency map that explains what regions are most salient to a given classifier, a CA approach tries to identify all possible salient regions for any hypothetical classifier (hence, classifier-agnostic). In other words, a CA approach may be inadequate for interpreting a specific classifier and is better suited for identifying salient regions for a class of image classification models.\" \n\nI'll focus now on studying how this paper addresses the 4 contributions they claim in the intro:\n\n> (1) We find that incorporating both masked-in classification maximization and masked-out entropy maximization objectives leads to the best saliency maps, and continually training the classifier improves the quality of generated maps. \n\nThere are no standard deviations reported from multiple runs so nothing can be statistically claimed. Also, looking at table 1 this is not clear. It seems MinClass works the best compared to almost all other methods in terms of PxAP. In terms of other metrics it seems like no best can be determined.\n\n> (2) We find that the masking model requires only the top layers of the classifier to effectively generate saliency maps. \n\nThis is reported in the table. It is not clear if this is a strong contribution as it would just be specific to this method and without a standard deviation we cannot conclude anything.\n\n> (3) Our final model outperforms other masking-based methods on WSOL and PxAP metrics. \n\nThe reported difference between the methods is 48.6 vs 48.4.\n\n> (4) We find that a small number of examples—as few as ten per class—is sufficient to train a masker to within the ballpark of our best performing model.\nThis is reported but the paper doesn't have a section detailing the experiments or showing how this number is derived.\n\n\nThe paper could be improved by refining the contributions and detailing what evidence should be observed to support these claims and then providing a significant amount of evidence. Right now the paper is not focused in general and does not focus on supporting the claims made in the introduction and therefore is not ready for publication.\t", "extra_scores": {}, "extra_text": {}}, {"review_id": "KODoHhOBji", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nBy the first look, this work itself does not introduce any new architecture or novel algorithm. It takes what is considered as the popular choices in generating classifier saliency masks, and conducts quite extensive sets of experiments to dissect the components by their importance. The writing is pretty clear in narrative and the experimental findings are surprising and significant. \n\nGood things to mention:\n\n1. The importance of using CA seems quite evident from Table 1.\n2. The importance of using multiple resolutions is also evident.\n3. The findings of few shot learning ability for masker is quite interesting, class diversity seems to play a very import role.\n\nSome questions:\n1. Given the fact that all experiments are done with ImageNet, which is widely considered as a solved task with almost super-human classifiers, do you think the superb performance of those classifiers could inflate your findings? Do you think the results in Table 1 and Figure 5 would still hold consistently when the classifier is not as performant?\n2. I am not sure I understand the point of the figure on the right of Figure 4. What is the take-way message from it?\n3. Compare (e,f,i,j) in Table 1, it seems I does not make a difference as long as O is used. What is the reason there?", "extra_scores": {}, "extra_text": {}}, {"review_id": "2u5QpP13al0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "/*************************Post-Rebuttal**********************/\nThe authors address many of my concerns well, and I agree with their rebuttal.\n\nThe modified manuscript also looks good, too.\n\nI raise my rating.\n\n/*************************Pre-Rebuttal**********************/\n\nPros.:\n1. The proposed method is simple and reasonable but better than the current state-of-the-art methods.\n2. The paper is clearly presented and well-organized.\n4. The authors conduct many ablation studies to validate the different variants of the proposed methods. \n5. The few-shot localization built in the proposed method is also interesting, and worth being explored.\n\n\nCons.:\n1. Limited Novelty\n\nThe proposed method is similar to [Ref. 1, 2], but there is no discussion in the related work. After reading this paper and [Ref. 1, 2], I don't see any significant difference in [Ref. 1, 2], the authors focus on top-down saliency detection, and use the masked image scheme to train the network. They proposed a CNN framework that contains two CNN modules, an image-level classifier and a pixel-level map generator. They also use the saliency from the map generator as the classifier's input and then optimize the classifier scores to derive the saliency map and train the generator. In this paper, the loss is L2 loss and binary-class setting, but I think the main idea is almost the same. Therefore, I think the novelty is very limited. I would like to see that the authors point the significant difference between the proposed method and [Ref. 1, 2] and give a detailed discussion.\n\n[Ref. 1] Hsu et al., Weakly Supervised Saliency Detection with A Category-Driven Map Generator, BMVC'17\n\n[Ref. 2] Hsu et al., Weakly Supervised Salient Object Detection by Learning A Classifier-Driven Map Generator, TIP'19\n\n2. No significant improvement\n\nCompared to Zolna et al. (2020), the improvement of the proposed method is not significant, 48.6 v.s. 48.4 in OM,  36.1 v.s. 35.8 in LE, 61.4 v.s. 62.0 in F1. I think this improvement is minor. \n\n3. About few-shot experiments\n\nThe few-shot experiments are interesting, but why aren't other methods conducted for these experiments? The authors claim that the few samples are sufficient to train the proposed method, but other methods may also require a small number of training samples to achieve similar performance. Besides, I am not clear why the proposed method could work well under the few-shot setting because there is no specific module or design for the few-shot setting.   Could the authors explain it in detail?\n\n4. Other issue:\n\nWhat is the difference between the proposed method and other weakly supervised or unsupervised object saliency detection, such as [Ref 3, 4, 5, 6, 7]? In these works, their goal is also to train a network to predict a class-agnostic saliency map and highlight the most salient object in the images under the unsupervised or weakly supervised setting. Therefore, I would like to know the major difference between the tasks. If they are similar to the solved problem, the discussion and comparison should be conducted.\n\n[Ref. 3] Zhang et al., Zhang. Supervision by fusion: Towards unsupervised learning of deep salient object detector, ICCV'17\n\n[Ref. 4] Li et al., Weakly supervised salient object detection using image labels, AAAI'19\n\n[Ref. 5] Wang et al., Learning to detect salient objects with image-level supervision, CVPR'17\n\n[Ref. 6] Zhang et al., Deep unsupervised saliency detection: A multiple noisy labeling perspective, CVPR'18\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the reviewers found parts of the paper interesting, the main concern about this paper was lack of novelty and marginal improvements obtained by the proposed methods.
Decision: Reject