Investigating and Simplifying Masking-based Saliency Methods for Model Interpretability

1 INTRODUCTION . The success of CNNs ( Krizhevsky et al. , 2012 ; Szegedy et al. , 2015 ; He et al. , 2016 ; Tan & Le , 2019 ) has prompted interest in improving understanding of how these models make their predictions . Particularly in applications such as medical diagnosis , having models explain their predictions can improve trust in them . The main line of work concerning model interpretability has focused on the creation of saliency maps–overlays to an input image that highlight regions most salient to the model in making its predictions . Among these , the most prominent are gradient-based methods ( Simonyan et al. , 2013 ; Sundararajan et al. , 2017 ; Selvaraju et al. , 2018 ) and masking-based methods ( Fong & Vedaldi , 2017 ; Dabkowski & Gal , 2017 ; Fong & Vedaldi , 2018 ; Petsiuk et al. , 2018 ; Chang et al. , 2019 ; Zintgraf et al. , 2017 ) . In recent years , we have witnessed an explosion of research based on these two directions . With a variety of approaches being proposed , framed and evaluated in different ways , it has become difficult to assess and fairly evaluate their additive contributions . In this work , we investigate the class of masking-based saliency methods , where we train a masking model to generate saliency maps based on an explicit optimization objective . Using a general formulation , we iteratively evaluate the extent to which recently proposed ideas in the literature improve performance . In addition to evaluating our models against the commonly used Weakly Supervised Object Localization ( WSOL ) metrics , the Saliency Metric ( SM ) , and the more recently introduced Pixel Average Precision ( PxAP ; Choe et al. , 2020 ) , we also test our final models against a suite of “ sanity checks ” for saliency methods ( Adebayo et al. , 2018 ; Hooker et al. , 2018 ) . Concretely , we make four major contributions . ( 1 ) We find that incorporating both masked-in classification maximization and masked-out entropy maximization objectives leads to the best saliency maps , and continually training the classifier improves the quality of generated maps . ( 2 ) We find that the masking model requires only the top layers of the classifier to effectively generate saliency maps . ( 3 ) Our final model outperforms other masking-based methods on WSOL and PxAP metrics . ( 4 ) We find that a small number of examples—as few as ten per class—is sufficient to train a masker to within the ballpark of our best performing model . 2 RELATED WORK . Interpretability of machine learning models has been an ongoing topic of research ( Ribeiro et al. , 2016 ; Doshi-Velez & Kim , 2017 ; Samek et al. , 2017 ; Lundberg et al. , 2018 ) . In this work , we focus on interpretability methods that involve generating saliency maps for image classification models . An overwhelming majority of the methods for generating saliency maps for image classifiers can be assigned to two broad families : gradient-based methods and masking-based methods . Gradient-based methods , such as using backpropagated gradients ( Simonyan et al. , 2013 ) , Guided Backprop ( Springenberg et al. , 2015 ) , Integrated Gradients ( Sundararajan et al. , 2017 ) , GradCam ( Selvaraju et al. , 2018 ) , SmoothGrad ( Smilkov et al. , 2017 ) and many more , directly use the backpropagated gradients through the classifier to the input to generate saliency maps . Masking-based methods modify input images to alter the classifier behavior and use the regions of modifications as the saliency map . Within this class of methods , one line of work focuses on optimizing over the masks directly : Fong & Vedaldi ( 2017 ) optimize over a perturbation mask for an image , Petsiuk et al . ( 2018 ) aggregates over randomly sampled masks , Fong & Vedaldi ( 2018 ) performs an extensive search for masks of a given size , while Chang et al . ( 2019 ) includes a counterfactual mask-infilling model to make the masking objective more challenging . The other line of work trains a separate masking model to produce saliency maps : Dabkowski & Gal ( 2017 ) trains a model that optimizes similar objectives to Fong & Vedaldi ( 2017 ) , Zolna et al . ( 2020 ) use a continually trained pool of classifiers and an adversarial masker to generate model-agnostic saliency maps , while Fan et al . ( 2017 ) identifies super-pixels from the image and then trains the masker similarly in an adversarial manner . Salient Object Detection ( Borji et al. , 2014 ; Wang et al. , 2019 ) is a related line of work that concerns identifying salient objects within an image as an end in itself , and not for the purpose of model interpretability . While it is not uncommon for these methods to incorporate a pretrained image classification model to extract learned visual features , they often also incorporate techniques for improving the quality of saliency maps that are orthogonal to model interpretability . Salient object detection methods that are trained on only image-level labels bear the closest similarity to saliency map generation methods for model interpretability . Hsu et al . ( 2017 ) and follow-up Hsu et al . ( 2019 ) train a masking model to confuse a binary image-classification model that predicts whether an image contains an object or is a ‘ background ’ image . Wang et al . ( 2017 ) apply a smooth pooling operation and a Foreground Inference Network ( a masking model ) while training an image classifier to generate saliency maps as a secondary output . Evaluation of saliency maps The wave of saliency map research has also ignited research on evaluation methods for these saliency maps as model explanations . Adebayo et al . ( 2018 ) and Hooker et al . ( 2018 ) propose sanity checks and benchmarks for the saliency maps . Choe et al . ( 2020 ) propose Pixel Average Precision ( PxAP ) , a pixel-wise metric for scoring saliency maps that accounts for mask binarization thresholds , while Yang & Kim ( 2019 ) create a set of metrics as well as artificial datasets interleaving foreground and background objects for evaluating the saliency maps . These works have shown that a number of gradient-based methods fail the sanity checks or perform no better than simple edge detectors . Hence , we choose to focus on masking-based methods in this paper . 3 MASKING-BASED SALIENCY MAP METHODS . We start by building a general formulation of masking-based saliency map methods . We take as given a trained image classifier F : x → y , that maps from image inputs x ∈ RH×W×C to class predictions ŷ ∈ [ 0 , 1 ] K , evaluated against ground-truth y ∈ { 1 · · ·K } . Our goal is to generate a mask m ∈ [ 0 , 1 ] H×W for each image x such that the masked-in image x m or the masked-out image x ( 1−m ) maximizes some objective based on output of a classifier given the modified image . For instance , we could attempt to mask out parts of the image to maximally deteriorate the classifier ’ s performance . This mask m then serves as a saliency map for the image x . Concretely , the per-image objective can be expressed as : arg min m λoutLout ( F ( x ( 1−m ) ; θF ) , y ) + λinLin ( F ( x m ; θF ) , y ) +R ( m ) , where Lout , Lin are the masked-out and masked-in objectives over the classifier output , λout , λin are hyperparameters controlling weighting of these two objectives , θF the classifier parameters , and R ( m ) a regularization term over the mask . The masked-in and masked-out losses , Lout and Lin , correspond to finding the smallest destroying region and smallest sufficient region as described in Dabkowski & Gal ( 2017 ) . Candidates for Lout include negative classification cross-entropy and prediction entropy . For Lin , the obvious candidate is the classification cross-entropy of the masked-in image . We set λin = 0 or λout = 0 if we only have either a masked-in or masked-out objective . The above formulation subsumes a number of masking-based methods , such as Fong & Vedaldi ( 2017 ) ; Dabkowski & Gal ( 2017 ) ; Zolna et al . ( 2020 ) . We follow Dabkowski & Gal , amortize the optimization by training a neural network masker M : x→ m , and solve for : arg min θM λoutLout ( F ( x ( 1−M ( x ; θM ) ) ; θF ) , y ) +λinLin ( F ( x M ( x ; θM ) ; θF ) , y ) +R ( M ( x ; θM ) ) , where M is the masking model and θM its parameters . In our formulation , we do not provide the masker with the ground-truth label , which differs from certain other masking-based saliency works ( Dabkowski & Gal , 2017 ; Chang et al. , 2019 ; Fong & Vedaldi , 2018 ) . In practice , we often desire model explanations without the availability of ground-truth information , so we focus our investigation on methods that require only an image as input . 3.1 MASKER ARCHITECTURE . We use a similar architecture to Dabkowski & Gal and Zolna et al .. The masker takes as input activations across different layers of the classifier , meaning it has access to the internal representation of the classifier for each image . Each layer of activations is fed through a convolutional layer and upsampled ( with nearest neighbor interpolation ) so they all share the same spatial resolution . All transformed layers are then concatenated and fed through another convolutional layer , upsampled , and put through a sigmoid operation to obtain a mask of the same resolution as the input image . In all our experiments , we use a ResNet-50 ( He et al. , 2016 ) as our classifier , and the masker has access to the outputs of the five major ResNet blocks . Figure 1B shows the architecture of our models . Following prior work ( Fong & Vedaldi , 2017 ) , we apply regularization on the generated masks to avoid trivial solutions such as masking the entire image . We apply L1 regularization to limit the size of masks and Total Variation ( TV ) to encourage smoothness . Details can be found in Appendix A.1 . Model OM ↓ LE ↓ SM ↓ PxAP ↑ . Train-Validation Set . Validation Set . 3.2 CONTINUAL TRAINING OF THE CLASSIFIER . Because neural networks are susceptible to adversarial perturbations ( Goodfellow et al. , 2015 ) , masking models can learn to perturb an input to maximize the above objectives for a given fixed classifier without producing intuitive saliency maps . While directly regularizing the masks is one potential remedy , Zolna et al . ( 2020 ) propose to train the masker against a diverse set of classifiers . In practice , they simulate this by continually training the classifier on masked images , retain a pool of past model checkpoints , and sample from the pool when training the masker . We adopt their approach and distinguish between a masker trained against a fixed classifier ( FIX ) and against a pool of continually trained classifiers ( CA , for Classifier-Agnostic ) . We highlight that saliency maps for FIX and CA address fundamentally different notions of saliency . Whereas a FIX approach seeks a saliency map that explains what regions are most salient to a given classifier , a CA approach tries to identify all possible salient regions for any hypothetical classifier ( hence , classifier-agnostic ) . In other words , a CA approach may be inadequate for interpreting a specific classifier and is better suited for identifying salient regions for a class of image classification models .

Summary: By the first look, this work itself does not introduce any new architecture or novel algorithm. It takes what is considered as the popular choices in generating classifier saliency masks, and conducts quite extensive sets of experiments to dissect the components by their importance. The writing is pretty clear in narrative and the experimental findings are surprising and significant.