Explaining Image Classifiers by Counterfactual Generation

1 INTRODUCTION . The decisions of powerful image classifiers are difficult to interpret . Saliency maps are a tool for interpreting differentiable classifiers that , given a particular input example and output class , computes the sensitivity of the classification with respect to each input dimension . Fong & Vedaldi ( 2017 ) and Dabkowski & Gal ( 2017 ) cast saliency computation an optimization problem informally described by the following question : which inputs , when replaced by an uninformative reference value , maximally change the classifier output ? Because these methods use heuristic reference values , e.g . blurred input ( Fong & Vedaldi , 2017 ) or random colors ( Dabkowski & Gal , 2017 ) , they ignore the context of the surrounding pixels , often producing unnatural in-filled images ( Figure 2 ) . If we think of a saliency map as interrogating the neural network classifier , these approaches have to deal with a somewhat unusual question of how the classifier responds to images outside of its training distribution . To encourage explanations that are consistent with the data distribution , we modify the question at hand : which region , when replaced by plausible alternative values , would maximally change classifier output ? In this paper we provide a new model-agnostic framework for computing and visualizing feature importance of any differentiable classifier , based on variational Bernoulli dropout ( Gal & Ghahramani , 2016 ) . We marginalize out the masked region , conditioning the generative model on the non-masked parts of the image to sample counterfactual inputs that either change or preserve classifier behavior . By leveraging a powerful in-filling conditional generative model we produce saliency maps on ImageNet that identify relevant and concentrated pixels better than existing methods . 2 RELATED WORK . Gradient-based approaches ( Simonyan et al. , 2013 ; Springenberg et al. , 2014 ; Zhang et al. , 2016 ; Selvaraju et al. , 2016 ) derive a saliency map for a given input example and class target by computing the gradient of the classifier output with respect to each component ( e.g. , pixel ) of the input . The reliance on the local gradient information induces a bias due to gradient saturation or discontinuity in the DNN activations ( Shrikumar et al. , 2017 ) . Adebayo et al . ( 2018 ) observed that some gradientbased saliency computation reflect an inductive bias due to the convolutional architecture , which is independent of the network parameter values . Reference-based approaches analyze the sensitivity of classifier outputs to the substitution of certain inputs/pixels with an uninformative reference value . Shrikumar et al . ( 2017 ) linearly approximates this change in classifier output using an algorithm resembling backpropagation . This method is efficient and addresses gradient discontinuity , but ignores nonlinear interactions between inputs . Chen et al . ( 2018 ) optimizes a variational bound on the mutual information between a subset of inputs and the target , using a variational family that sets input features outside the chosen subset to zero . In both cases , the choice of background value as reference limits applicability to simple image domains with static background like MNIST . Zintgraf et al . ( 2017 ) computes the saliency of a pixel ( or image patch ) by treating it as unobserved and marginalizing it out , then measuring the change in classification outcome . This approach is similar in spirit to ours . The key difference is that where Zintgraf et al . ( 2017 ) iteratively execute this computation for each region , we leverage a variational Bernoulli distribution to efficiently search for optimal solution while encouraging sparsity . This reduces computational complexity and allows us to model the interaction between disjoint regions of input space . Fong & Vedaldi ( 2017 ) computes saliency by optimizing the change in classifier outputs with respect to a perturbed input , expressed as the pixel-wise convex combination of the original input with a reference image . They offer three heuristics for choosing the reference : mean input pixel value ( typically gray ) , Gaussian noise , and blurred input . Dabkowski & Gal ( 2017 ) amortize the cost of estimating these perturbations by training an auxiliary neural network . 3 PROPOSED METHOD . Dabkowski & Gal ( 2017 ) propose two objectives for computing the saliency map : • Smallest Deletion Region ( SDR ) considers a saliency map as an answer to the question : What is the smallest input region that could be removed and swapped with alternative reference values in order to minimize the classification score ? • Smallest Supporting Region ( SSR ) instead poses the question : What is the smallest input region that could substituted into a fixed reference input in order to maximize the classification score ? Solving these optimization problems ( which we formalize below ) involves a search over input masks , and necessitates reference values to be substituted inside ( SDR ) or outside ( SSR ) the masked region . These values were previously chosen heuristically , e.g. , mean pixel value per channel . We instead consider inputs inside ( SDR ) or outside ( SSR ) the masked region as unobserved variables to be marginalized efficiently by sampling from a strong conditional generative model1 . We describe our approach for an image application where the input comprises pixels , but our method is more broadly applicable to any domain where the classifier is differentiable . 1 Whereas ( Zintgraf et al. , 2017 ) iteratively marginalize single patches conditioned on their surroundings , we model a more expressive conditional distribution considering the joint interaction of non-contiguous pixels . Consider an input image x comprising U pixels , a class c , and a classifier with output distribution pM ( c|x ) . Denote by r a subset of the input pixels that implies a partition of the input x = xr ∪ x\r . We refer to r as a region , although it may be disjoint . We are interested in the classifier output when xr are unobserved , which can be expressed by marginalization as pM ( c|x\r ) = Exr∼p ( xr|x\r ) [ pM ( c|x\r , xr ) ] . ( 1 ) We then approximate p ( xr|x\r ) by some generative model with distribution pG ( xr|x\r ) ( specific implementations are discussed in section 4.1 ) . Then given a binary mask2 z ∈ { 0 , 1 } U and the original image x , we define an infilling function3 φ as a convex mixture of the input and reference with binary weights , φ ( x , z ) = z x+ ( 1− z ) x̂ where x̂ ∼ pG ( x̂|xz=0 ) . ( 2 ) 3.1 OBJECTIVE FUNCTIONS . The classification score function sM ( c ) represents a score of classifier confidence on class c ; in our experiments we use log-odds : sM ( c|x ) = log pM ( c|x ) − log ( 1− pM ( c|x ) ) . ( 3 ) SDR seeks a mask z yielding low classification score when a small number of reference pixels are mixed into the mask regions . Without loss of generality4 , we can specify a parameterized distribution 2 zu = 0 means the u-th pixel of x is dropped out . The remaining image is xz=0 . 3 The infilling function is stochastic due to randomness in x̂ . 4 We can search for a single mask z′ using a point mass distribution qz′ ( z ) = δ ( z = z′ ) over masks qθ ( z ) and optimize its parameters . The SDR problem is a minimization w.r.t θ of LSDR ( θ ) = Eqθ ( z ) [ sM ( c|φ ( x , z ) ) + λ‖z‖1 ] . ( 4 ) On the other hand , SSR aims to find a masked region that maximizes classification score while penalizing the size of the mask . For sign consistency with the previous problem , we express this as a minimization w.r.t θ of LSSR ( θ ) = Eqθ ( z ) [ −sM ( c|φ ( x , z ) ) + λ‖1− z‖1 ] . ( 5 ) Naively searching over all possible z is exponentially costly in the number of pixels U . Therefore we specify qθ ( z ) as a factorized Bernoulli : qθ ( z ) = U∏ u=1 qθu ( zu ) = U∏ u=1 Bern ( zu|θu ) . ( 6 ) This corresponds to applying Bernoulli dropout ( Srivastava et al. , 2014 ) to the input pixels and optimizing the per-pixel dropout rate . θ is our saliency map since it has the same dimensionality as the input and provides a probability of each pixel being marginalized ( SDR ) or retained ( SSR ) prior to classification . We call our method FIDO because it uses a strong generative model ( see section 4.1 ) to Fill-In the DropOut region . To optimize the θ through the discrete random mask z , we follow Gal et al . ( 2017 ) in computing biased gradients via the Concrete distribution ( Maddison et al. , 2017 ; Jang et al. , 2017 ) ; we use temperature 0.1 . We initialize all our dropout rates θ to 0.5 since we find it increases the convergence speed and avoids trivial solutions . We optimize using Adam ( Kingma & Ba , 2014 ) with learning rate 0.05 and linearly decay the learning rate for 300 batches in all our experiments . Our PyTorch implementation takes about one minute on a single GPU to finish one image . 3.2 COMPARISON TO FONG & VEDALDI ( 2017 ) . Fong & Vedaldi ( 2017 ) compute saliency by directly optimizing the continuous mask z ∈ [ 0 , 1 ] U under the SDR objective , with x̂ chosen heuristically ; we call this approach Black Box Meaningful Perturbations ( BBMP ) . We instead optimize the parameters of a Bernoulli dropout distribution qθ ( z ) , which enables us to sample reference values x̂ from a learned generative model . Our method uses mini-batches of samples z ∼ qθ ( z ) to efficiently explore the huge space of binary masks and obtain uncertainty estimates , whereas BBMP is limited to a local search around the current point estimate of the mask z . See Figure 5 for a pseudo code comparison . In Appendix A.1 we investigate how the choice of algorithm affects the resulting saliency maps . To avoid unnatural artifacts in φ ( x , z ) , ( Fong & Vedaldi , 2017 ) and ( Dabkowski & Gal , 2017 ) additionally included two forms of regularization : upsampling and total variation penalization . Algorithm 1 : BBMP ( Fong & Vedaldi , 2017 ) Input image x , classifier score sM , sparsity hyperparameter λ , objective L ∈ { LSSR , LSDR } Initialize single mask z ∈ [ 0 , 1 ] d while loss L is not converged do Clip z to [ 0 , 1 ] Compute reference image x̂ heuristically Compute in-fill φ ( x , z ) = x · z + x̂ · ( 1− z ) With φ , compute L by Equation 4 or 5 Update z with∇zL end while Return z as per-feature saliency map Algorithm 2 : FIDO ( Ours ) Input image x , classifier score sM , sparsity hyperparameter λ , objective L ∈ { LSSR , LSDR } , generative model pG Initialize dropout rate θ ∈ [ 0 , 1 ] d while loss L is not converged do Sample minibatch z ∈ { 0 , 1 } d ∼ Bern ( θ ) Sample reference image x̂ ∼ pG ( x̂|z , x ) Compute in-fill φ ( x , z ) = x · z + x̂ · ( 1− z ) With φ , compute L by Equation 4 or 5 Update θ with∇θL ( Maddison et al. , 2017 ) end while Return θ as per-feature saliency map Figure 5 : Pseudo code comparison . Differences between the approaches are shown in blue . Upsampling is used to optimize a coarser θ ( e.g . 56 × 56 pixels ) , which is upsampled to the full dimensionality ( e.g . 224 × 224 ) using bilinear interpolation . Total variation penalty smoothes θ by a ` 2 regularization penalty between spatially adjacent θu . To avoid losing too much signal from regularization , we use upsampling size 56 and total variation as 0.01 unless otherwise mentioned . We examine the individual effects of these regularization terms in Appendices A.2 and A.4 , respectively .

Summary: This paper aims to find important regions to classify an image. The main algorithm, FIDO, is trained to find a saliency map based on SSR or SDR objective functions. The main novelty of this work is that it uses generative models to in-fill masked out regions by SSR or SDR. As such, compared to existing algorithms, FIDO can synthesize more realistic samples to evaluate.