GANMEX: Class-Targeted One-vs-One Attributions using GAN-based Model Explainability
Authors: Sheng-Min Shih
Pin-Ju Tien
Zohar Karnin
Venue:  (2021)
Keywords: Deep Neural Networks
Attribution Methods
Generative Adversarial Networks
Interpretability
Explainability

Abstract: Attribution methods have been shown as promising approaches for identifying key features that led to learned model predictions. While most existing attribution methods rely on a baseline input for performing feature perturbations, limited research has been conducted to address the baseline selection issues. Poor choices of baselines can lead to unfair attributions as well as limited ability of one-vs-one explanations for multi-class classifiers, which means explaining why the input belongs to its original class but not the other specified target class. Achieving one-vs-one explanation is crucial when certain classes are more similar than others, e.g. two bird types among multiple animals. One-vs-one explanations focus on key differentiating features rather than features shared across the original and the target classes. In this paper, we present GANMEX, a novel algorithm applying Generative Adversarial Networks (GAN) by incorporating the to-be-explained classifier as part of the adversarial networks. Our approach effectively selects the baseline as the closest realistic sample belong to the target class, which allows attribution methods to provide true one-vs-one explanations. We showed that GANMEX baselines improved the saliency maps visually and led to stronger performance on perturbation-based evaluation metrics over the existing baselines. Attribution results with the existing baselines are known to be insensitive to model randomization, and we demonstrated that GANMEX baselines led to better outcome under the randomization sanity checks.

Under review as a conference paper at ICLR 2021
GANMEX: ONE-VS-ONE ATTRIBUTIONS GAN-BASI MODEL EXPLAINABILITY
USING
Anonymous authors Paper under double-blind review
ABSTRACT
Attribution methods have been shown as promising approaches for identifying key features that led to learned model predictions. While most existing attribu- tion methods rely on a baseline input for performing feature perturbations, lim- ited research has been conducted to address the baseline selection issues. Poor choices of baselines limit the ability of one-vs-one explanations for multi-class classifiers, which means the attribution methods were not able to explain why an input belongs to its original class but not the other specified target class. Achiev- ing one-vs-one explanation is crucial when certain classes more similar than others, e.g. two bird types among multiple animals, by focusing on key differenti- ating features rather than shared features across classes. In this paper, we present GANMEX, novel approach applying Generative Adversarial Networks (GAN) by incorporating the to-be-explained classifier as part of the adversarial networks. Our approach effectively selects the baseline as the closest realistic sample belong the target class, which allows attribution methods to provide true one-vs-one explanations. We showed that GANMEX baselines improved the saliency maps and led to stronger performance on perturbation- evaluation metrics over the existing baselines. Existing attribution results are known for being insensitive to model randomization, and we demonstrated that GANMEX basclines led to better outcome under the cascading randomization of the model.
INTRODUCTION
Modern Deep Neural Network (DNN) designs have been advancing the state-of-the-art, performance of numerous machine learning tasks with the help of increasing model complexities, which at the same time reduces model transparency. The need for explainable decision is crucial for earning trust of decision makers, required for regulatory purposes Goodman & Flaxman (007)), and extremely useful for development and maintainability.
Due to this, various attribution methods were developed to explain the DNNs decisions by attributing importance weight to each input feature. In high level, most attribution methods, such as inte- grated gradient (IG) (Sundararajan et al. (2017), DeepSHAP (Lundberg & Lee (2017)), Deeplift (Shrikumar al. (2017)) and Occlusion (Zeiler Fergus (2013)), alter the features between the original values and the values of some baseline instance, and accordingly highlight the features that impacts the model's decision. While extensive research has been conducted on the attribution algo- rithms, research regarding the selection of baselines is rather limited, and it is typically treated as an afterthought. Most existing methodologies by default apply uniform- iaffurn-valle baseline, which can dramatically impact the validity of the feature attributions (Sturmfels et al. (2020)) and as result, existing attribution methods showed rather unperturhed output even after complete randomization of the DNN (ddbaao et al. (2018)).
In multi-class classification setting, existing baseline choices do not allow specifying a target class, and this has limited the ability for providing class-targeted or one- os-vee-one explanation, meaning explaining why the input belongs to class A and not a specific class B. These explanations are crucial when certain classes are more similar than others, often happens for example when the classes have a hierarchy among them. For example, classification task of apples, oranges and bananas, model decision for apples oranges should be based their color rather than the shape

---

Under review as a conference paper at ICLR 2021
since both an apple and orange are round. This would intuitively only happen when asking for an explanation of why apple and not orange' rather than 'why apple'.
In this paper, present GAN-based Model EXplainability (GANMEX), novel methodology for generating one-vs one explanations leveraging GAN. In nutshell, we use GANs to produce a base- line image which is realistic instance from target class that resembles the original instance. A naive use of GANs can be problematic because the explanation generated would not be specific to the to-be-explained DNN. We lay out well-tuned recipe that avoids these problems by incorporat- ing the classifier as static part of the adversarial networks and adding a similarity loss function for guiding the generator. We showed in the ablation study that both swapping in the DNN and adding the similarity loss are critical for resulting the correct explanations. To the best of our knowledge, GANMEX is the first to apply GAN for explaining DNN decisions, and furthermore the first to provide a realistic baseline image, rather than an ad-hoc null instance.
We showed that GANMEX baselines can be used with a variety of attribution methods, including IG, DeepLIFT, DeeP and Occlusion, to produce one-vs-one attribution superior compared with existing approaches. GANMEX outperformed the existing baseline choices on perturbation- evaluation metrics and showed more desirable behavior under the sanity checks of randomizing DNNs. Other than its obvious advantage for one-vs-one explanations, we show that by replacing only the baselines and without changing the attribution algorithms,, GANMEX greatly improves the saliency maps for binary classifiers, where one-vs-one and one-vs-all are equivalent.
RELATED WORKS
2.1 ATTRIBUTION METHODS AND SALIENCY MAPS
Attribution methods and their visual form, saliency maps, have been commonly used for explaining DNNs. Given input [x1, ...,, xN| RN and model output S(x) (s.......(((( E RCC an attribution method for output assign contribution to each pixel Ag,c = (01,-..ay, There are two major attribution method families: Local attribution methods that are based on infinitesimal fea- ture perturbations, such as gradient saliency (Simonyan et al. (2014)) and gradient*input (Shrikumar al. (2066)) and global attribution methods that based on feature perturbation with respect to baseline input (Ancona et al. (018)). ) We focus on global attribution methods since they tackle the gradient discontinuity issue local attribution methods, and they are known be more effective on explaining the marginal effect of a feature's 's existence (Ancona et al. (2018)). In this paper, we discussed five popular global attribution methods below:
Integrated Gradient (IG) (Sundararajan et al. (017)) calculates a path integral of the model gra-
dient from baseline image J to the input image x: IG; (ar; .;)) f S( a(x -). The baseline is commonly chosen to be the zero input and the integration path is selected as the straight path between the baseline and the input.
DeepLIFT Shrikumar et al. (211 addressed the discontinuity issue by performing backpropaga- tion and assigns score Caz,Ar At to each neuron in the networks based on the input difference the baseline Ar; 24 3. and the difference in the activation to that of the baseline Al = ((x) - (()). that satisfies the nummation-to-dertt
property Σ, CAR,AL At.
Occlusion (Zeiler& & Fergus (2013); Ancona et al. (2018)) applies full-feature perturbations by re- moving each feature and calculating the impacts on the DNN output. The feature removal was performed by replacing its value with zero, meaning an all zero input was implicitly used as the baseline.
DeePSHAP (Chen et al. (2019); Lundberg & Lee (007)) Shrikumar et al. (217)) was built upon the framework of DeepL.FTT but connecting the multipliers of attribution rule (rescale rule) to SHAP values, which are computed by 'erasing features'. 'he operation of erasing one or more features require the notion of a background, which is defined by either a distribution (c.g. uniform distribution over the training set) or single baseline instance. For practical reasons, it is common to choose single baseline instance to avoid having to store the entire training set in memory.
Expected Gradient (Erion et al. (019%) is variant of IG that calculates the expected attribution over a prior distribution of baselmee input, usually approximated by the training set X2, meaning
2

---

Under review as a conference paper at ICLR 2021
EG; = -g= (0. S( + a(x - #)) where U is the uniform distribution. In other words, the baseline of IG is replaced with a uniform distribution over the samples in the training set.
A crucial property of the above methods is their need for baseline, which is either explicitly or implicitly defined. what follows we show that these methods are greatly improved by modifying their baseline to that chosen by GANMEX.
2.2 THE BASELINE SELECTION PROBLEM
Limited research has been done on the problem of baseline selection so far. simple "most natural input", such as zero values of all numerical features is commonly chosen as the baseline. For image inputs, uniform images with all pixels set to the max/min/mmeiium values are commonly chosen. The static baselines frequently cause the attribution area where the feature values are different from the baseline values, and hide the feature importance where the imput values are close the baseline values (Sundararajan & Taly (2018) Adebayo et al. 2018); Kindermans et al. (2017); Sturmfels et al. (2020)..
only focus on or even overly highlight the
Several none-static baselines have been proposed in the past, but each of them suffered from its own downsides (Sturmlels et al. (2020)). Fong & Vedaldi (017)) used blurred images as baselines, but the results are biased toward highlighting high-frequency information from the input. Bach al.) (2013) make use of the training samples by finding the training example belonging to the target class closest to the input in Euclidean distance. Even though the concept of minimum distance is highly desirable, but in practice, the nearest neighbor selection in high dimensional space can frequently lead to poor outcome, and most of the nearest neighbors are rather distant from the original input.
Along the same concept, expected gradient simply samples over all training instances instead of identifying the closest instance (Erion et al.] (2019). Expected gradient benefits from ensembling in way similar to that of SmoothGrad, which averages imposing Gaussian noise on the original image (Smilkov et al. (2017); Hooker et al.] (2019). We claim however that averaging over the training set does not solve the issue; for example, due to the foreground being located in different sections of the images, the average image would often resemble uniform baseline.
multiple saliency maps produced by
2.3 ONE-VS-ONE AND ONE-VS-ALE VLL ATTRIBUTION
In multi-class settings, while one-vs-all explanation As,e, (x) was designed to explain why the input belong to its original class Co and not the others, one-vs-one explanations aim provide an attribution As,co (x) E RN that explains why x belong to c. and not the specified target class G. Most existing attribution methods were primarily designed for one-vs-all explanation, but was proposed to extend to one-vs-one by simply calculating the attribution with respect to the difference of the original class probability to the target class probability Sun(z)) Se, Se, (z) (Bach al. 2013); Shrikumar et al. (017)).
is easy to think of examples where this somewhat naive formulation will not provide correct one-vs-one explanation. Taking the example of fruit classification, for both apples and oranges the explanation could easily be the round shape, and taking the difference between those will result in an arbitrary attribution. We claim that without taassttrgeete bascline, the modified attributions will still omit the "vs-one" aspect of the one-vs-one explanation. Take IG for example, As As.ai(()) = As,c, (x) As,e, (I)). With zero baseline, the target class score Sc, (x) and its gradient will likely stay close to zero along the straight path from the input to the zero baseline, meaning that As,c, (x) ze because the instance never belongs the target class. With this in mind the one-vs-one explanation is not very informative with respect to the target class ct.
Few class-targeted baselines were proposed in the past. The minimum distance training sample (MDTS) described in Section 2.2 is class-targeted as the sample was selected from the designated target class. While the original expected gradient was defined for one-vs-all explanation only, we extended the method to one-vs-one by sampling the baselines only from the target class. However, as mentioned in Section 2.2 MDTS is frequently hindered by the sparsity of the training set in the high dimensional space, and expected gradient suffers from undesired effects caused by uncorrelated training samples. The problem of baseline selection, especially for the one-vs-o-e oe-ev explainability, has
3

---

Under review as a conference paper at ICLR 2021
presented challenging problem, because the ideal baseline choice can simply be absent from the training set.
2.4 GAN AND BAGGE-TO-IMAGE TRANSLATION
Image-to-Image Translation is family of GAN originally introduced by Isola et al. (2017) for creating mappings between two domains of data. While the corresponding pairs of images are rare in most real-world dataset, Zhu et al. (2017) has made the idea widely applicable by introducing a reconstruction loss to tackle the tasks with unpaired training dataset. Since then, more efficient and better performing approaches have been developed improve few-shot performance (Liu al. 22019) and output diversity (Choi et al. (202)). Nevertheless, we found the StarGAN variant proposed by Choi al. (2017) specifically applicable to the baseline selection problem because of its standalone class discriminator in the adversarial networks as well as the deterministic mapping that preserve the styles of the translated images (Choi et al. (2020)). GANs have not been applied for explaining DNNs in the past to our best knowledge.
Prior to our work. Chang et al. (0118) proposed the fill-in the dropout region (FIDO) methods and suggested generators including CA-GAN Yu et al. (2018) for filling in the masked area. However, the CA-GAN generation was designed for calculating the smallest sufficient region and smallest de- stroying region Dabkowski & Gal (2017) that only produced 1-vs-all explanations. FIDO is compu- tationally expensive as an optimization task required for each attribution map. The fill-in method requires unmasked area for reference, hence only works for small subset of attribution meth- ods. More importantly, the FIDO is highly dependent on the generator's 's capability of recreating the image based on partially masked features. With pre-trained generators like CA-GAN, we argue that the resulting saliency map is more associated with the pre-trained generator instead of the classifier itself.
GAN-BASED MODEL. EXPLAINABILITY
It has been previously established that attribution methods are more sensitive to features where the input values are the same as the baseline values and less sensitive to those where the input values and the baseline values are different (Adebayo et al. 2018): ; Sundararajan & Taly (2018)) Therefore, we expect a well-chosen baseline to differ from the input only on the key features. Good candidates for achieving this would be the sample in the target class but with minimum distance the input.
Formally, for a one-vs-one attribution problem -s, (I), We define the class-targeted baseline to be the closest point in the input space (not limited to the train set) that belongs the target class
Be, (x) = arg min || 2|
(1)
sEG.,
Here, Ge, is the set of realistic examples in the target class, and ||- is the Euclidean distance. By using this baseline we have As,c.-ae (I. Be, (x)) providing the explanations as to why input r belongs to its original class e, and not class c.. Now, since isn't realistic to optimize within the actual set Go, we work with softer version of Equation [ Ba, (ar) arg minseR (Ilæ
āll - log T(ū, c.).. where T(ã, 0t) represent the probability of ã belonging to the target class, meaning ã E Ge.- Given classifier S(z)
(z), .... Sc(x)), we have the estimate Sz()) to the probability of a realistic image ă to be in class c. In order to make use of this we decompose T(ë, ()) R( (z) where R(ă) indicates the probability of f being a realistic image. We end up with the following objective for the baseline instance.
Be, (1)) = arg min (IL=- =1| log R(t) log Se, (i))
BERN
3.1 APPLYING STARGAN TO THE CLASS-TARGETE BASELINE
Here we introduce GaN-based Model EXplainability (GANMEX) that uses GAN to generate the class-targeted basclines. Given input and target class c,, GANMEX aims to generate class- targeted bascline G(r, ce) that achieve the three following objectives:
1. The baseline belongs to the target class (with respect to the classifier).
2. The baseline is realistic sample.
4

---

Under review as a conference paper at ICLR 2021
Class
Decision boundry
Class
Area representing realistic samples
Area representing training datasets
(a)
Input
(b)
Figure 1: Intuition of using GANs for generating class-targeted baselines in SVHN dataset. Without GANs, closest target class sample easily be unrealistic (a). while the GAN helps confine the sample in the realistic sample space (h). The MDTS baseline and other training samples used in expected gradient can be very different from the input (c). (d) shows the zero baseline that is the most commonly used.
3. The baseline is close to the input.
To further explain the need for all 3 objectives, we point the reader to Figure I The GANMEX baseline represents the "closest and realistic target class baseline". Without the assistance of GANs, the selected baseline can easily either fall into the domain of unrealistic image. A naive fix will choose a realistic image from the training set, but that will not be close to the input. Finally, for correct one-vs-on one explainability we need the baseline to belong to a specific target class. We have provided more intuitions behind the baseline selection requirements in Appendix G
We chose StarGAN (Choi et al. (0177) as the method for computing the above T' rather R function. Although many Image-to to- Image translation methods could be applied to do so, StarGAN inherently works with multi-class problems, and allows for natural way of using the already trained classifier S as discriminator, rather than having train different discriminator.
StarGAN provides a scalable image-to-image translation approach by introducing (1) single gen- erator G(a,, e) accepting an instance x and a class c, and producing a realistic example in the target class c, (2) two separate discriminators: (z) for distinguishing between real and fake images, and Dets (z, e) for distinguishing whether belongs to class c. It introduced following loss functions
L
E, (og((Dno + Eg, [log(1 Dane (G(x, c)))
C Ber (c((
C/ Londl
(5)
Dre Ee,c',ree G(G(, c),))
(6)
Here, E. defines the average over the variables in the subscript, where æ is an example in the training set, and c' are classes. Ladv is the standard adversarial loss function between the generator and the discriminators, Ligs and C/1, are domain classification loss functions for real images and fake images, respectively, and Liee is the reconstruction loss commonly used for unpaired image-to-image translation to make sure two opposite generation action will lead to the original input. The combined loss functions for the generator and the discriminator are
LD -Lur + Nor
La = Ludy + Nag C/ + Neehhee
(8)
The optimization procedure for StarGAN alternates between modifying the discriminators Date (I), Dess (z, c) to minimize CD. and the generator to minimize Lg.
Equation 8 is almost analogical to equation The term Ladv corresponds to log R(ã)) and the term X Aar C. corresponds to the term log Se, (z). There is mismatch between the term \ Cree and || - 21I. One forces the generator to be invertible, while the other forces the generated image to be close to the original. We found that the Liee term is useful to encourage the convergence of the GAN. However, a similarity term || - G(z,c) c) is also needed in order for the baseline image to be close to the origin- this allows for better explainability. We show in what follows (Figure BB
5

---

Under review as a conference paper at ICLR 2021
(A)
Baselines
Integrated Gradient Exp.
DeepLIFT
Occlusion
DeepSHAP
Input
Grad
Zero
MOTS
0
6
6
MNIST
1->4
1
f 4
3
5 5 9
3-25
2
8
6
40
16
46
0->6
SVHN
3->5
58
39
horse
CIFAR10
dog airplane
deer
>> ship
(B)
Input
baseline
DF
Ozc
Input baseline IG
DF
Occ
Input baseline
IG
DF
Occ
(7.9)
9
(9,31
(airplane ship)
(2.7)
2
7
(2,7)
(truck, airplane)
(0.6)
0
(9,2)
(horse,
deer)
(6,8)
3
8
(7,91
(cat,
dog)
Figure 2: (A) Saliency maps for multi-class datasets (MNIST, SVHN, CIFAR10) generated with various base- lines, including zero baseline (Zero), MDTS and GANMEX, with classes c. - Ct indicated for each plc. (B) Mis-classification analysis showing with the (mis-classified class, correct class) pairs. The baseline columns show the expected images generated by GANMEX for the correct classes, and the saliency maps show the explanation of "why not the correct class" produce by IG, DeepLIFT (DL) and occlusion (Oec).
Appendix D) that without this similarty term, the created image indeed be farther away from the origin. Other than the added similarity term, for GANMEX we replace the discriminator (as(c(ā) with the classifier (T), since mentioned above, this way the generator provides baseline adapted to our classifier. Concluding, we optimize the following term for the generator
c/ log(1 (+)( Xyn log(S.())) A G(), Asmmm 2112
(9)
where short for G(x, Notice that we used I.1 distance rather than I.2 for the similarity loss, because I.2 distance leads to blurring outputs for image-to-imant translation algorithms (Isola iet al. (2017)). Other image-to-image translation approaches can potentially select baselines satisfy- ing the criteria (2) and (3) above, but they lack the replaceable class discriminator component, that crucial for explaining the already trained classifier. We provide several ablation studies in Ap- pendix D where show that without incorporating the no-be-explained classifier the adversarial networks. the GAN generated baselines will fail the randomization sanity checks. We provide more implementation details including hyper-paramene in Appendix A.2

---

Under review as a conference paper at ICLR 2021
(A)
(D)
MNIST
MINIT
(G)
10
Input
08
Score cett
0.
Orig
Output
FC
CNN2
CNNI
0.4
C
Baseline
GAMMEX baseline
(B)
200
400
900
800
(E)
IntGrad
SVHN
SVHN
0.
0.8
DeepLIFT
Score data
06
06
Occlusion
0.-
-02
Expected
(C)
200
800
(F)
Gradient
0.8
06
IntGrad
65
Sore dota
Spassty
64
Zero baselina
03
DeepLIFT
0.2
62
611
Occlusion
0.
100rod
Intorad GAMMEX
DespUIT
Displrt GAMMES
Occlusionn GAMMEX
IntGrad
HnGradd GANMEX
Expected
Expected Gradient
DeepUFT
Octlasion
Occlusion GANMEX
Figure 3: (A-C) Certurhation-basee evaluation plots for MNIST and SVHN, respectively. The dashed lines represent the non-class-targeted baselines and the solid lines represent class-targeted baselines. (D-F) Gini indices, with the yellow hars represent saliency maps with zero haselines and the green bars represent that of GANMEX baselines. (G) Sanity checks showing the original saliency maps (Orig) and saliency maps under cascading randomization over the four layers: output layer (Output), fully connected (FC). and two CNN layers (CNNI, CNN2).
4 EXPERIMENTS
In what follows we experiment with the datasets MNIST (LeCun & Cortes (010)), Street-Viee House Numbers (SVHN) (Netzer al. (201T)), CIFARIO Krizhevsky 2009) and apple2orange (huu et al. (2017)) Further details about the datasets and classifiers are given in Appendix A.
Our techniques designed to improve any global attribution method by providing improved baseline. In our experiments we consider four attribution methods IG, DeepLIFT, Occlusion, and DeepSHAP. The baselines we consider include the zero baseline (the default baseline in all four methods), minimum distance training sample (MDTS), and the GANMEX baseline. We also com- pared our results with modified version of expected gradient aimed to provide 1-vs-1 explanations, which runs IG over randomly chosen target class image from the training set, as opposed to random image from the training set.
4.1 ONE-VS-ONE ATTRIBUTION FOR MULTI-CLASS CLASSIFIERS
We tested the one-vs-one attribution on two multi-class datasets MNIST, SVHN, and CIFARIO. As shown in Figure A, the GANMEX baseline successfully identified the closest transformed image in the target class as the baseline. Take explaining why 0 and not 6 for example, the ideal baseline would keep the "C"'-hape part unchanged, and only crase the top-right corner and complete the lower circle, which was achieved by GANMEX. Limited by the training space, MDTS baselines were generally more different from the input image. Therefore. the explanation made with respect to GANMEX baselines line and expected gradient. We observed the same trends across more numbers, where GANMEX helps IG, DIF,,.IFT, Occlusion and DeepSHAP disregard the common strokes between the original
more focused on the key features compared to that of the MDTS base-
7

---

Under review as a conference paper at ICLR 2021
and targeted digits, and focusing only the key differences. The out-performance of GANMEX was even more obvious in the SVHN datasets, where the numbers can have any font, color, and background, and in CIFARIO, which has more complexity and diversity. Notice that both the zero baseline and training set baseline cause the explanation to have more focus on the background, and in contrast, the GANMEX example focuses only on the key features that would cause the digit to change.
Zero baselines, on the other hand, were generally unsuccessful in making one-vs-one explanations. The attributions on MNIST look similar to the original input and ignores everything in the back- ground, and the attributions on SVHN were rather noisy. As shown in Figure [8] attributions based on zero baselines only changed marginally with different target classes. This shows that purposely designed class-targeted baselines are required for meaningful one-vs-one explanation.
cissclassification Analysis We next demonstrated how one-vs-one saliency maps can be used for trouble-shooting miscclassification cases. For an input that belongs to class Co but was mis- classified as class Cm-- As.cm (2) = As,em-te. (z, B., (z)) provides explanation to why æ be- longs to Cm and not Co with respect to the trained classifier, and this will help human understand how the classifier has led to the incorrect decision. We provided examples in Figure 2 where the samples were mis-classified. For MNIST and SVHN samples, the cisscaassification mostly hap- pened when the digits were presented in non-typical way. The GANMEX baseline Be, (x) show how more typical digit should have been written according the trained classifier, and the attri- bution As.em-te (x) highlights the area that led to the mis- cissclassiiication. CIFARIO presented more complex classification challenges. and the classifier can easier confused ship with an airplane be- cause of the pointy front and the lack of sea horizon, or a dog with cat because of the shape of the ears and the nose, and those areas were highlighted in the one-vs-one saliency maps.
Perturbation-base evaluation We followed the perturbation- evaluation suggested by Bach et al. (2015) that flips input features starting from the ones with the highest saliency values and evaluates the cumulative impacts on the score delta S., - Sc, as proposed by /Shrikumar et al. (017). Flipping feature means to provide with value of 1 x where x is its original value, assuming all features are normalized to x E [o, 1]. A wanted behavior from the attribution map is that the score delta will decrease as rapidly as possible as we flip the features one by one. We provide in Figure A-C the perturbation curves for both MNIST, SVHN and CIFAR 10, plotting the score delta function of the number of flipped features. It is painfully clear that that by using GANMEX baseline rather than the alternative zero baseline. the descent of the curve is much faster, meaning that we successfully capture the most important features using GANMEX. This holds true for all attribution methods. As side note, notice that in SVHN when we flip all features the score delta goes back to where it was in the beginning as opposed to going down to zero. This is due to the fact that once all features are flipped, we are back to having the same digit as before.
Gini-index We calculated the Gini Index representing the sparseness of the saliency maps as pro- posed by Chalasani et al. (218)., where a larger score means sparser saliency map, which is a desired property. 3 shows experiments comparing the different techniques with zero base- line vs. GANMEX. Other than IG/SVHN where GANMEX has a visible advantage, the results are roughly the same; we suspect that the sparseness of zero baseline attribution was benefited from incorrectly hiding key features, as shown in Figure 2 hand, consistently under-performs its counterpart (IG+GANMEX) in both datasets.
A and 3
A-C. Expected gradient, on the other
Cascading randomization We performed the sanity checks proposed by Adebayo et al.) (2018) that performs cascading randomization from top to bottom layers of the DNN and observe the changes in the saliency maps. Specifically, layer by layer we replace the model weights with Gaus- sians random variables scaled to have the same norm. For meaningful model explanations, we would expect the attributions to be gradually randomized during the cascading randomization process. In contrast, unperturbed saliency maps during the model randomization would suggest that the attribu- tions were based on general features of the input and not specifically based on the trained model.
Figure G shows the experiment on MNIST data with the network layers named (input to output) CNNI, CNN2, FC, Output. It shows that even though the saliency maps generated by the original IG, IIe and Occlusion were rather unperturbed (still showing the shape of the digit) after the
8

---

Under review as a conference paper at ICLR 2021
Baselines
Integrated Gradient
DeepLIFT
Input
Blur
GANMEX
Zero
Max
Blur
GANMEX
Zero
Max
Blur
GANMEX
Figure 4: Saliency maps for the classifier on the apple2orange datasct with four baseline choices: zero bascline (Zero), maximum value baseline (Max), blurred baseline (Blur), and GANMEX baseline (GANMEX).
model randomization, with the help of GANMEX, both the baselines and the saliency maps were perturbed over the cascading randomization. Expected gradient, while showing more randomization compared to the zero bascline saliency maps, still roughly shows the shape of the digit throughout the sanity check.
4.2 ATTRIBUTION FOR BINARY CLASSIFIERS
In addition to the one-vs-one aspect, GANMEX generally improves the quality of the saliency maps compared with the existing baselines, and this can be tested on binary datasets where the one-vs-one explanations and the one-vs-all explanations apples and oranges both have round shapes but have different colors, so we would expect the saliency maps on reasonably performing classifier to highlight the colors of the fruit, but not the shapes, and definitely not the background.
equivalent. For apple2orange dataset, conceptually
In Figure 4 and Figure 9 we compared the saliency map generated by DeepLIFT and IG with the zero input, max input, blurred image, with those generated with the GANMEX baselines. In all non-GANMEX baselines (Zero, Max, Blur) we commonly observe one of errors in the saliency map. The first eтrоr consists of highlighting the background. The second highlights only the edge of the apple(s) providing the false indication that the model is basing its decisions on the shape of the object rather than its color. It is quite clear that neither of these errors occur when using the GANMEX baselines as the background never present and the full shape of the apple())(orang(()) is highlighted.
5 CONCLUSION AND FUTURE WORK
We have proposed GAN-based model explainability, a novel approach for generating one-vs-one ex- planation baselines without being constrained by the training set. We used the GANMEX baselines in conjunction with IG, DeepLIFT, SHAP, and Occlusion, and to our surprise, the baseline replace- ment was all takes to address the common downside of the existing attribution methods (blind to certain input values and fail to randomize with the model randomization) and significantly improve the one-vs-one explainability. The out-performance was demonstrated through perturbation- evaluation, sparseness measures, and cascading randomization sanity checks. The one-vs-one expla- nation achieved by GANMEX opens up possibilities for obtaining more insights about how DNNs differentiate similar classes.
While GANMEX showed promising results on explaining binary classifiers, where one-vs-all and one-vs-one explanations are directly comparable. open questions remain on how apply GANMEX to one-vs-all explainability for multi-class classifiers, and how to best optimize the GAN component to effective generate baselines for classification tasks with large number of classes.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gareett (eds.), Advances in Neural Information Processing Systems 31, nips. cc/ppperc/
pp. 9505-9515. Curran Associates, Inc., 2018. URL httt:://papppeee
8160 1000-sannty--oo checks- for saliency maps mnf
David Alvarez-Melis and Tommi S. Jaakkola. Towards robust interpretability with self-explaining neural networks, 2018.
Marco Ancona, Enea Ccolini, Cengiz Öztireli, and Markus Gross. Towards better understand- ing of gradient-based attribution methods for deep neural networks. In International Confer- ence on Learning Representations, 2018. URL http:::/ppeereview.cett net.eeww.eeeeeeeee.eeeeeeeeeeee forumm/
SY21R9JAM
Sehastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robed Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):1-46, 07 2015. doi: 10.137 httj/ooounaaloooonc 000.e00.31100
URL /attts::/doi.orooe http://ooo.ooo/////////o 1371/ journal pone. 0130140
Prasad Chalasani, Jiefeng Chen, Amrita Roy Chowdhury, Somesh Jha, and Xi Wu. Concise ex- planations of neural networks using adversarial training. CoRR, abs/1810.06583, 2018. URL.
http://arxiv wttp:///.rrbbb//11000..0
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image classifiers by adaptive dropout and generative in-filling. CoRR, 0ass1807.08024, 2018. URL
http:://rxiis org/ hbtp:///a/111800222
Hugh Chen, Scott Lundberg, and Su-In Lee. Explaining models by propagating shapley values of local components, 2019.
Yunjey Choi, Min-Je Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo.
Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. CoRR, abs/1711 2017.
wttp/or////////// 09020
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, 2020.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30, pp. 6967-6976. Curran Asso- ciates, Inc., 2017. URL https: 0060ef47b 21 210111160b9999 60b91 91309988822288818 2452
Gabriel G. Erion, Joseph D. Janizek, Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Learn- ing explainable models using attribution priors. CoRR, abs/1906.00670, 2019. URL http:
10670
Ruth Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. CoRR, abs/I 704.03296, 2017. Utl.Atttp.//aarr...orr/////....0 03296
Bryce Goodman and Seth Flaxman. European union regulations on algorithmic decision-making and a "right to explanation". Al magazine, 38(3):50-57, 2017.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretabil- ity methods in deep neural networks, 2019.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CORR, Abs//1044.04811, 2017. URL http: www./oorg/aaee abs/
T704 04861
10

---

Under review as a conference paper at ICLR 2021
P. Isola, J. Zhu, T. Zhou, and A. A. Efros. Image-to-imann translation with conditional adversarial networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 5967-5976, 2017.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. arXiv preprint arXiv:1711.00867, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. UtLhttp://yann:
lecun. boom/exdb/mmae
Ming- Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz.
Few-shot unsupervised image-to-image translation. CoRR, abs/1905.01123, 2019. URL http:
wawxiv..rr/abb/II00000 01723
Scott M Lundberg and Su-In Lee. dictions. pp. 4765-4774, 2017.
unified approach interpreting model pre-
URL http://papers wwwps.ccppppp...........cp
7062
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning, 2011.
An phi Nguyen and María Rodríguez Martínez. On quantitative aspects of model interpretability, 2020.
W. Samek, A. Binder, G. Montavon, Lapuschkin, and Müller. Evaluating the visualization of what deep neural network has learned. IEEE Transactions on Neural Networks and Learning Systems, 28(11):2660-26 2673, 2017.
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just a black box: Learning important features through propagating activation differences. CoRR, 010.00133,. 2016. URLhttpp /ttp://xxxxii..oooo/m/
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through
propagating activation differences. CoRR, abs/1 704.02685, 2017. URL.ttp::/arxxi..o abs/1704, 02685
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, abs/1312.6034, 2014. URL
attp://arxiv: hbtgg//bs/1331 6034
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. Smooth- grad: removing noise by adding noise. CORR, abs/1706.033 778253325, 2017. UtL.pttp://arxiv
org/abs/1706, 03825
J.T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for simplicity: The all convolutional net. In ICLR (workshop track), 2015. URL http://lmb. informatik.
uni fww.iurg.gg.cmm htt.:ugg....eeullcciiiiiooooo 201.//011151111/0
Pascal Sturmfels, Scott Lundberg, and Su-In Lec. Visualizing the impact of feature attribution baselines. Distill, 2020. doi:
Mukund Sundararajan and Ankur Taly. note about: Local explanation methods for deep neural
networks lack sensitivity to parameter values. CoRR, abs/1 (os/1886.04205, 2018. URL http://
arxiv.org/aae wtw.l/..rg//b////..... 0420) 04205
Mukund Sundararajan, Ankur Taly, and Qigi Yan. Axiomatic attribution for deep networks. CoRR, abs/1703.01365, 2017. URL //arxiv wtreiiiiorrr///eooo abs/170........ 01365
Richard Tomsett, Dan Harborne, Supriyo Chakraborty, Prudhvi Gurram, and Alun Preece. Sanity checks for saliency metries, 2019.
11

---

Under review as a conference paper at ICLR 2021
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. Generative image inpainting with contextual attention, 2018.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. CoRR, abs/1311.2901 2013. URL http:////aaaaa..rrr///////111111
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image trans-
lation using cycle-consistent adversarial networks. CoRR, 1059933 2017. http:
//arx/v.orc//ab//700...
12

---

Under review as a conference paper at ICLR 2021
A IMPLEMENTATION DETAILS
A.1 DATASETS AND CLASSIFIERS
MNIST (LeCun & Cortes (0100) The classifier consists of two 6x6 CNN layers with a stride of 2, followed by 256-unit fully connected layer, a dropout layer with p = 0.5, and the 10 output neu- As shown in (Springenberg et al. (2015) the stride 1 CNN achieved comparable performance with pooling layers. The classifier was trained for 50 epochs and achieve a test accuracy of 99.3%.
Street- View House Numbers (SVHN) (Netzer et (2011) We tested our models the cropped version of SVHN and used the same model architecture with that of MNIST and achieved a test accuracy of 90.3% after 50 epochs of training.
CIFARIO Krizhevsky (2009) We trained a classifier consist of 4 repetitive units, with each unit constructed by 3x3 CNN layers and 2x2 average pooling layer, with each CNN layer followed by batch normalization layer. The classifer achieved 87.8% test accuracy after 100 epochs of training.
apple2orange (Zhu et al. (2017) We trained a classifier taking the original 256x256 image as in- put. The classifier was constructed by adding global average pooling layer on top of MobileNet (Howard et al. (2017)), and then followed by a dense layer of 1024 neurons and a dropout layer of p = 0.5 before the output neurons. The classifier was trained for 50 epochs and achieve a test accuracy of 87.7%.
A.2 BASELINE GENERATION WITH GANMEX
Our baseline generation process based on StarGAN (Choi al. (2017)) We used the Tensorflow- GAN implementation htttp://gittubbcoo//enns/tt (Equation D.:
and made the following two modifications
1. The class discriminator Dats is replace by the target classifier S to be explained.
2. A similarity loss Lxim is added to the training objective function.
We train the GANMEX model for 100k steps for the MNIST and apple2orange datasets, 300k steps for the SVHN dataset, and 400k steps for the CIFARIO dataset. Only the train split is used for training, and the attribution results and evaluation were done on the test split of the dataset.
A.3 ATTRIBUTION METHODS
We used DeepExplain (https://github.co (ttps://gicuub..o///arrroooo hwtpp///w......ooooo///e...aaaaa for generating saliency maps with IG, DeepLIFT, and Occlusion. We modified the code base to use the score delta (S S instead of the original class score (S.,) ) and allowing replacing the zero baseline (see Section 2. by custom baselines from GANMEX and MDTS. Expected gradient was separately implemented according to the formulation in Erion et al. (2019). We set the number sampling steps to 200 for both IG and Expected Gradient, and used Occlusion-1 that only perturb the pixel itself (as supposed to perturbing the whole neighboring patch of pixels).
The DeepSHAP saliency maps calculated using SHAP We made similar modification to replace the original class score by the score delta and feed in the custom baseline instances.
In all saliency maps shown in the paper, blue color in indicates positive values and red color indicates negative values. We skipped Occlusion for large images (apple2orange) and also skipped SHAP for full dataset evaluations due to the computation resource constraints.
B BASELINE DISTANCE ANALYSIS
To measure how various baseline selection approaches satisfy the minimum distance requirements in Equation I calculated D(x, ž) ||ا ø| for (1) GANMEX, (2) MDTS, (3) randomly selected sample in the target class as baseline and (4) zero baseline. GANMEX was on-par with MDTS on the MNIST dataset, but on SVHN and CIFAR dataset that have more degrees of freedom (object
13

---

Under review as a conference paper at ICLR 2021
(A)
(
2500
2000
GANMEX: mean = 1.50
5
Count
1500
1000
MDTS: mean 3.84
500
0
4
6
10
edge
(C)
(D)
(E)
Input GAN
MDTS
Input
GAN 3.0
MDTS
Input
GAN
MDTS
1.0
3.1
5.1
2.2
6.0
1->9
8->2
2->9
1.0
2.1
3.0
8.1
1.7
6.0
2
5
B
58
0
9
3->2
5->8
0->9
1.0
3.6
3.0
7.2
1.8
6.0
2
2
2
5
70
78
2->3
2->5
0->6
1.0
4.3
3.0
8.1
1.7
6.0
7
28
88
7->9
2->8
4->7
Figure 5: (A) Vertical edge
SVHN image. (B) Histogram of sample baseline distance (D (1r, i))
in the vertical edge area. (C-E) Samples comparing GANMEX and MDTS baselines with Dodge 3) indicated on the top baseline images.
(C) Easy cases GANMEX (D) (z, 2) 1). (D) Difficult cases
GANMEX (D (x, .) zs 3). (E) Difficult cases for GANMEX (T) (x,5) RJ 6).
Table 1: Baseline distance analysis comparing the average intra-class distance (Dimra sampled) and the average inter-class distance (Diner, sampled), with the average distance from the instance to the bascline input generated by GANMEX (GAN), MDTS, random selection (RAND), and zero inputs (Zero).
Data Size
Avg. Distance
Baseline Distance
Dimension
Train
Test
D
Diteer
GAN
MDTS
Rand
Zero
MNIST
784 (28 28 x 1)
60,000
10,000
8.96
10.32
7.17
7.18
10.32
9.28
SVHN
3072 3)
73,257
26,032
14.42
14.48
3.42
5.94
15.44
26.52
CIFARIO
3072 (32 x 3)
50.000
10,000
18.28
19.06
10.54
19.01
29.04
14

---

Under review as a conference paper at ICLR 2021
size, color, orientation, background, ...), GANMEX was significantly better in identifying minimum distance baselines compared to the ia-sample search. The high dataset complexity of was supported by the average intra-class distance, the average distance between any two instances within the same class, which was higher than that of MNIST. Note that the resulting sample to baseline distance (GAMME)) is much higher in MNIST than in SVHN, because there were more boundary values (Os and Is) in MNIST.
We further evaluated the similarity distance on the vertical edge area (D (zz ā)) of the SVHN im- ages (Figure5) A). Empirically, we observed that the digits of interest were rarely present in the ver- tical edge, and therefore, we would expect a closest baseline choice will lead to minimal D (z. ă) under the minimum distance requirements. We provided a histogram in Figure [ for comparing the distribution of Dodg ;) for MDTS and GANMEX, and we presented sampled success/failure cases in Figure 5 C-F. Overall, GANMEX leads to baselines that are closer to the original samples.
C ADDITIONAL METRICS
Here we evaluated the different baseline choices with additional metrics (Table 2). AOPC, r. mea- sures the area over the perturbation curve within the first perturbation steps (Samek et al. (2017) 7):
Tomsett et al. (2019). One potential downside of AOPC, is that the metric is only sensitive to the top L features in the saliency map and not the rest. Therefore, in addition to AOPC we calculated AOPCall the area over the perturbation curve across all feature. The gradient family of IG and expected gradient generally outperformed DeepL.FTT and occlusion on the AOPC metrics, with IG+GANMEX performed the best overall.
The sparsity (Chalasami al. (2018)) measured by the Gini index is a desirable property for one- vs-one attribution. We expect good one-vs-one explanation to highlight only the differentiating features. Compared with one-vs-all saliency maps, one-vs-one saliency maps should highlight a smaller subset of features, especially when the target classes are similar to the original classes. Therefore, one would expect more sparse one-vs-one saliency maps are more likely to be correct. Our results showed that saliency maps generated by IG+GANMEX and IG+MDTS have higher Gini index and therefore are more sparse compared to other methods (Table
We also measured the faithfulness reported by Alvarez-Melis & Jaakkola 20188: Tomsett et al. (2019) and monotonicity suggested by phi Nguyen & Martínez (2020). Instead of measuring the cumulative effect of alternating set of features, both faithfulness and monotonicity measure the impacts on alternating single features. We found that expected gradient and Occlusion+zero baseline are the best performers on those two metrics.
Lastly, we designed an inverse localization metric to measure whether the explanation is localized in the focus area. We observed that the digits mostly have aspect ratios, meaning that their widths are smaller then their heights. As a results, the areas at the two vertical edges are generally not covered by the primary numbers, and instead, they usually show the background or the neighboring numbers. Therefore, we can reasonably expect the saliency map sensitivity to be location in the center area (area excluding the vertical edges), and not the vertical edge area 5
Based on this observation, we designed the inverse localization metric, I(A(x)) (ard Eiesuue (z)))) (card)/ camre Eies. (x)).. that calculates the ratio the average absolute sensitivity between the vertical edge area and the center area. Scenee and Sodge represent the feature set in the center area and the edge area, respectively. card(.) measures the cardinality of feature sets, and x and A(r) are the sample and the corresponding saliency map. A lower L(A(z)) would mean that the saliency map A(r) is more localized in the focus area.
As shown in Table 2 we see consistent trend of the saliency maps with GANMEX baselines being more localized (lower inverse localization) compared to MDTS baselines across all attribution methods, and expected gradient and the zero baselines generally lead to the worst results. The saliency maps produced by occlusion+GANMME was the most localized among all the methods tested.
(lower
To summarize, we evaluated multiple attribution methods and baseline combinations with metrics that assess different properties of the saliency maps. While the results of faithfulness and monotinic- ity were less consistent, GANMEX has consistently lead to better metrics on inverse localization,
15

---

Under review as a conference paper at ICLR 2021
Table 2: (GAN).
Additional metrics for attribution methods using the zero baseline (Zero), MDTS, and GANMEX
Integrated Gradient
BG
DeepLIFT
Occlusion
Metries
Dataset
7ero
MI)TS
GAN
7ero
MISSS
GAN
7ero
MDTS
GAN
1OPC100
MIST
0.614
1.249
1.421
1.260
0.505
0.639
0.724
0.705
1,050
1.221
SVHN
0,346
0.861
0.921
0.878
0.377
0.634
0.621
0.317
0.547
0.549
CIFARIO
0.298
0.485
0.494
0,516
0.323
0.464
0.451
0.441
0.492
0.440
AOPCat
MIST
0,889
1.098
1.263
1.13
0.859
0.933
0.992
0.877
1.019
1.114
SVHN
0.564
0.844
0,822
0.626
0.649
0.750
0.751
0.528
0.586
0.585
CIFARIO
0.696
0.788
0.808
0.789
0.726
0.780
0.794
0.628
0.694
0.706
sparsity
MIST
0.909
0.911
0.919
0.827
0.047
0.047
0,046
0.062
0.058
0.058
SVHN
0.606
0.713
0.783
0.615
0.131
0.133
0.139
0.168
0.139
0.144
CIFARIO
0.565
0.639
0.626
0.522
0.164
0.171
0.169
0.325
0.260
0.223
faithfulness
MIST
0.182
0.224
0,280
0.407
0.075
0.003
0.031
0.257
0.254
0.291
SVHN
0.017
0.265
0.270
0.548
-0,041
0,075
0.017
0.007
0.306
0.243
CIFARIO
0,005
0,028
0,027
0.054
0.003
0.017
0,017
0.288
0.285
0.225
monolonicity
MIST
0.118
0.196
0.264
0.357
0.087
0,150
0.206
0.244
0.239
0.280
SVHN
0.129
0.212
0.248
0.340
0.095
0,150
0.182
0.057
0.210
0.211
CIFARIO
0.008
0.050
0.042
0.058
0.004
0.044
0.033
0.175
0.140
0.087
inv. loculization
SVHN
0,268
0.128
0,113
0.217
0.268
0.156
0.123
0.268
0.144
0.100
AOPCL, AOPCall and sparsity, especially when compared with the non-class -targete zero baseline that were widely used in the field.
D ABLATION STUDIES
Here we analyzed the possibilities of using other GAN models. Different from StarGAN, most other tange-to-image translation algorithms do not have a stand-alone class discriminator that can be swapped with trained classifier. To simulate such restrictions, we trained similar GAN model but with the class discriminator trained jointly with the generator from scratch. 7 A shows that while the stand-alons GAN yields similar baseline with GANMEX, both the baselines and saliency maps of the stand-alone GAN remains unperturbed under cascading randomization of the model. This indicates that the class-wise explanations provided by stand-alone GAN were not specific to the to-be-explained classifier.
The importance of the similarity loss in Equation can be demonstrated on coNOTed-MNIST dataset, where we randomly assigned the digits with one of the three colors (red, green, blue}, with labels of the instances remain unchanged from the original MNIST labels of {0, 9.. The classifier was trained with the same model architecture and training process as for MNIST.
9}.
The dataset demonstrated different modes (colors in this case) that are irrelevant to the labels, and we would expect the class- targeted baseline for x would be another instance that has the same color as x. Figure B shows that the similarity loss the crucial component for ensuring that the baseline has the same color with the input. Without the similarity loss, the generated baseline instance can casily have a different color with the original image. The reconstruction loss itself does not provide the same-mode constraint because a mapping of G(red) - green and G'green) red does not get penalized by the reconstruction loss. While the reconstruction loss was not required for GANMEX and the same- same-mode constraint, we observed that some degrees of reconstruction loss help GANs converge faster.
E HYPER-PARAMETER ANALYSIS
We tested how the generated baselines change with respect to the hyperparemeters the GANMEX loss function. The prree-pprameters X/ar Aree and Asm-- presented in Equation control the degrees of the classification loss, reconstruction loss, and similarity loss, respectively. We performed the syper-parameter scan on the SVHN dataset as has enough complex and yet simple enough for visually assessing the attribution.
16

---

Under review as a conference paper at ICLR 2021
(A)
Input
(B)
Stand-alone GAN
Input
S+R
S
R
NA
Orig
Output
FC
CNN2 CNN1
3
8
8
3
8
6
6
Baseline
3
3
3
3
IntGrad
5
6
6
6
6
Occlusion
6
S
5
5
5
DeepLIFT
4
7
7
7 7
GANMEX
Orig Output
CNN2 CNN1
7
4
4
6
Baseline
IntGrad
Occlusion
DeepLIFT
Figure 6: (A) Cascading randomization baselines generated by stand-alone GAN lead little randomiza- tion on the saliency maps. (B) Colored-MNISS dataset. GAN basclines generated with both similarity loss and reconstruction loss (S+R), similarity loss only (S), reconstruction loss only (R), and none of those (NA). Only S+R and S successfully constrained the baselines in the same modes (colors) with the inputs.
17

---

Under review as a conference paper at ICLR 2021
Table 3: Run Time Analysis comparing the attribution computation time for IG, expected gradient (EG), DeepLIFT (DL.) and Occlusion (Occ), as well as the baseline generation time for EG, GANMEX (GAN), and MDTS. The computation was performed on single Tesla V100 GPU, and the compute time was measure in seconds calculation
all samples for the dataset, and the baseline generation time is the additional compute time on top of the attribution methods. ($) We selected same sampling number for IG and EG, and the baseline selection time of EG was estimated by the complexity difference of EG and IG. (f) The attribution inference time for CIFAR 10 was measured in 10 separate batches due to the memory constraint. ($) MDTS scarch was performed on CPU instead of GPU.
Attribution Inference
Baseline Generation
GAN Training
Dataset
Size
Dim.
IG
BG
DI.
Occ
EG8
GAN
MDTS
Steps
r (hour)
MNIST
10k
784
43.5
64.1
0.6
75.7
20.6
92.5
850.8
100k
5.2
SVHN
26k
3072
557.3
658.9
1.9
4917.0
101.6
399.8
3674.4
300k
18.2
CIFARIOF
10k
3072
239.4
294.8
30.0
1532.1
55.3
959.7
1085.8
400k
23.8
Classification Loss (N Low classification loss tended make some transformation unsuccessful, and high classification loss introduced additional noise that make the images unrealistic.
Similarity Loss (Aece) Similarity loss is the key component for minimum distance optimization. As we have shown in Section D and Figure BB at zero similarity loss, the generator is only constraint by the reconstruction loss and lead incorrect font colors and background. High similarity loss, on the other hand, makes the baselines to be too similar to the original images.
Reconstruction Loss (Asmm) ) As have mentioned in Section and Figure 7 reconstruction loss is not required for GANMEX, but it slightly helps GAN to converged. In contrast, high reconstruc- tion loss lead to incorrect outputs.
F COMPUTE TIME ANALYSIS
In Table 3 measure the GANMEX compute time compared with various attribution methods. While the GAN component takes 5-23 hours to train depending on the datasets, the inference step only requires one single forward operation, and the compute time (MNIST: 9.3 ms, SVHN: 15.4 ms, CIFAR10: 96.0 ms) is at the same order with IG and Occlusion. More experiment details are provided in the caption of Table 3
G INTUITIONS BEHIND THE MINIMUM DISTANCE REQUIREMENTS
Here we present the intuitions behind the baseline selection criteria from Section 3.J using simpli- fied formulation. Assuming a transformation p projecting from a set of high-level concept variables V to a sample z, with p(V), and we can separate V into three groups V = {Vrdis / V ). Here, d are the discriminating variables that leads to the model decision, for example, the color the fruits in apply/orange dataset; //omn are the contingent variables that independent the model decision but correlate with how the discriminating variables are presented (eg. size and loca- tion of the fruits); yiu are the irrelevant variables that are both independent the model decision and uncorrelated to the presentation of the discriminating variables (eg. background colors of the images). The expected one-vs-all explanation under the formulation would be
= (p(V)) = A(p(Vdii youn, V²r)) = ac., (Vdis /ron Ac,
(10)
with a being a transformation from the underlying concept variables to the explanation. Here we assumed correct mapping should be independent of / because the variable set has no impact on the discriminating variables themselves or how discriminating variables are presented.
Now if we apply the concept of one-vs-one, expect a one-vs-one explanation to produce
Aco (xr, (x:) -Ceg (Vdis Vddis /oon)
(11)
18

---

Under review as a conference paper at ICLR 2021
(A) Input 5
6
Input
Target
Target
0
2
3
4
6
7
8
9
0
1
2
4
5
7
8
0,55
0: 01:
01
5:
4
6:
71
5: 5
8
|
3
4
5
8
19
0: 01:
2: 5:
5:
6:
0:
5:
5
86
02
13
14
5
7
18
19
Ka==
0: f1:
2:
5:
4:
6:
71
81
5
80
1
12
13 16
1
1
18
19
55
0: 11:
2:
5:
4:
6:
5: 9
|
12
13
14
15
17
18
11
0 11:
D:
5:
4:
6:
|
8: 3
0
i|
12
33
14
15
37
18
20
0:
2:
3:
4:
6:
7: 8:
9
8
1
12
33
14
15
3
8
19
55.=50
0:
2:
3:
4:
6:
7::
8: 9
8
1 2
3
4
5
8
19
1,,=100
0:
8:
91
0
2
1
(B)
Target
Target
1
2
3
6
7
8
9
0
2
3
4
8 99
9
Noc=0
0:
1
2:
21
41
8:
0
2
4
5
A 1
0: 5:
2:
5:
4:
6
7:
5:
9
0
1
1/
13
14
66
1
8
"
0: 0:
0:
5:
4:
6:
71
8:
5
0
1 f |2
33
14
5
7
88
39
11
0: 11:
2:
5:
4:
6:
5:
9
0 1 1 12
33
14
5
17
88
9
20
0: 11:
D:
5:
4:
6:
71:
8:
0 i
12
13
14
5
7
88
9
500
0: :: D:
5:
5:
6:
5:
8:
5
0 H
33
14
6
7
88
9
A 100
0: 01
2:
5:
5:
5:
0
5:
9
6
M
l2
3
16
16
7
18
1
(C)
Target
Target
3
8
9
0
4
0: 4:
5:
41
6
7:
5: 9
3
1
33
4
5
8
39
5m
0: 8:
0:
3:
4:
6:
71
8:
9
30
1
1
13
1
15
1
8
99
Aam 10
0: |::
2:
5:
4:
6:
#:
5:
9
0
| l
12
13
14
5
1
88
3
Aam 20
0: |1:
0:
5:
4:
6
5:
9
0 | f
12
33
14
35
17
18
19
Amm =50
0: :
D:
5:
4:
6:
7:
5: 9
0
1 1 12
33
14
15
1
18
19
3:
9
10
||
12
A6
30
10
Amm
100
Figure 7: GANMEX baselines generated with various weights for the (A) classification loss, (B) similarity loss, and (C) reconstruction loss.
19

---

Under review as a conference paper at ICLR 2021
where Aco- and Be, were defined in Section and Valie and V are the discriminating variables for class c, and oç. In the apple orange example, Vidia would represent red color, and Vdds would represent orange color.
For baseline gerenative function (x) c., (p(Vais Vom, vin) = p(Vilin 7eon, ViT where Vdis Vrom and Vir are the concept variables for the generated baseline input. We can write out
Acg (x, Be,(a))))) Aco (p(Vdis, Voon Vir), p fromn (iir))
(12)
Although Acg-e, can be designed be independent of Vcon and Vcon we still need additional constraint to make Equation 12 satisfy the form of Equation IT While one can apply additional
constraints Acg-e, to satisty such requirements, an alternative approach would be requiring Ba, to ensure the following.
Frdis
V
(13)
Zcon
//con
(14)
piir
Vir
(15)
Equation 1 requires the baseline to belong to the target class ct, and this implies that without additional constraints to Acg-tee+ class- class-targeted one-vs-one baseline is required for correct one-vs- one explanations.
Equation 14 and Equation 13 combined have led to the closest input requirements in Section 3.1
Assuming a smooth transformation (p), minimizing the distance of I| B., (x)]l provides an ef- fective way of ensuring Equation 14 and Equation [5] Going back the apple/orange example, baseline satisfying Equation 13 15 for an apple image input would be an image with an orange fruit of the same size, the same location, with the background the original input, and all of the above can be achieved by the minimum distance sample described in Section 3.1
H ADDITIONAL FIGURES
20

---

Under review as a conference paper at ICLR 2021
(A) input
O
Target label
1
2
3
4
6
8
9
Avg
GANMEX baseline
/
2
3
4
5
C
7
8
9
IntGrad
IntGrad+GANMEX
DeepLIFT
DeepLIFT+GANMEX
DeepSHAP
DeepSHAP4GANMEX
(B)
Target label
input
0
2
4
6
8
9
Avg
GANMEX baseline
IntGrad
IntGrad+GANMEX
DeepLIFT
DeepLIFTHGAMMEX
DeepSHAP
DeepSHAP4GANMEX
Figure One-vs-one saliency maps using class targete baselines (GANMEX) vs non-class- targeted baselines (zero baselines). One-vs-one saliency maps generated using zero baselines show almost the same attributions regardless of the target class, making the one-vs-one saliency maps (columns with target lahels) similar to the one-vs-all saliency maps (the "Avg" columns that show the averaged saliency maps over all target classes). GANMEX baselines corrected the hehavior for both IG, DeepL.FT and DeepSHAP, and produced different attributions depending target classes.
21

---

Under review as a conference paper at ICLR 2021
Baselines
Integrated Gradient
DeepLIFT
Input
Blur
GANMEX
Zero
Max
Blur
GANMEX
Zero
Max
Blur
GANMEX
Figure 9: Additional examples saliency maps for the classifier on the apple2orange dataset with four baseline choices: zero baseline (Zero), maximum value baseline (Max), blurred baseline (Blur), and GANMEX baseline (GANMEX).
22


Reviews:
[{"review_id": "V6wirPRcQFw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper looks to use GANs to generate baselines for attribution methods. The focus on one-vs-one feature importance explanations is novel, to the best of my knowledge. The paper attempts to make progress on the baseline selection problem that has plagued the feature importance community. \n\nStrengths\n- As far as I know, the authors' contribution of one vs one attribution (compared to one vs any attribution) is novel. Whilst other works have alluded to this or ran heusristic experiments, this paper does a good job of formalizing the notion.\n- The ability for GANMEX to live on top of any other attribution method makes it an attractive addition to existing attribution methods. Thank you for visualizing the baselines generated by GANMEX, quite helpful :)\n\nWeaknesses\n-  A computational complexity analysis is required to gauge the practical utility of generating baselines with GANMEX. Also, it would be nice to give complexity of GANMEX compared to FIDO, EG, and simple nearest neighbor baselines. \n- In addition to the visual comparisons provided, it would have been helpful to evaluate explanations using exisiting evaluation criteria in the attribution literature (i.e., faithfulness, sensitivity, monotonicity, etc.) This paper has the opportunity to broadly assess the effects of various baselines on attributions.\n\nQuestions\n- While GANs seem like an attractive choice of deep generative model (DGM) for this problem, can you comment on or experiment with other DGMs (i.e., VAEs or specifically VAEACs [1])? However, any DGM that has latent class separation should suffice. You would be able to perform optimization in the latent space [2, 3, 4] and achieve similar class separation, as described in Figure 1.\n- The attributions in Figure 3.E seem like noise, while zero baseline seems visually appealing -- can you provide some intuition for why this occurs? The GAN feels like overkill for MNIST, but might be suitable for other high dimensional problems wherein the baseline needs to pick up on small nuances in the data.\n\n[1] https://openreview.net/forum?id=SyxtJh0qYm\n\n[2] https://arxiv.org/abs/1806.08867\n\n[3] https://arxiv.org/abs/2006.06848\n\n[4] https://arxiv.org/abs/1807.08024", "extra_scores": {}, "extra_text": {}}, {"review_id": "RQF2_GVImA0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Summary:\n\nThis paper considers the less-explored baseline selection issue in attribution methods for one-vs-one explanations of multi-class classifiers. The key insight is to construct the closest and realistic target class baseline. To this end, an existing image-to-image translation GAN model, namely StarGAN, is leveraged to transform an input example to another example in a target class yet is close to the input. This baseline can be integrated with a variety of attribution methods, including integrated gradient, DeepLIFT, Occlusion, and deepSHAP, and shows consistent improvements over zero baseline and minimum distance training sample for one-vs-one explanations. The experiments are conducted on three datasets – MNIST, SVHN, and apple2orange.\n\nPaper Strengths:\n\nThis paper addresses an important yet overlooked baseline selection problem. The way the authors address this problem is interesting by leveraging GAN models. Empirical evaluations demonstrate the effectiveness and generalizability of the proposed approach.\n\nPaper Weaknesses:\n\n1) The main weakness of this paper to me is the evaluation section. The proposed approach is only validated on simple datasets like MNIST and SVHN. It would be more convincing to show the effectiveness of the proposed approach on natural images and a large number of classes, like CIFAR and ImageNet, as used in the previous work such as IG.\n\n2) Following the comment in 1), the prior works, such as IG and DeepLIFT, have been used to analyze other types of models and been evaluated on other types of data, such as genomics and neural machine translation. In addition to images, would the proposed approach also apply to these domains?\n\n3) As illustrated in Figure 1, the key assumption of the proposed approach is that a GAN model (StarGAN) is able to generate examples that are much closer to the input examples than the training examples (i.e., minimum distance training sample). Under what conditions would such an assumption hold?\n\n4) Following the comment in 3), it would be interesting to show and analyze some failure cases.\n\n5) In the proposed approach, the StarGAN directly uses the already trained model classifier as its discriminator. How if the StarGAN trains its own discriminator without using the model classifier?\n\n6) It would be interesting to show the hyper-parameter (different trade-off lambdas) sensitivity.\n\n7) I understood that the authors focused on one-vs-one explanations. But I am interested to hear the authors’ thoughts on how to extend the proposed approach to one-vs-all explanations.\n\nAfter Rebuttal: \n\nI thank the authors for the rebuttal. I have also read the other reviewers’ comments. Unfortunately, the rebuttal is unconvincing and sometimes vague. I keep my original rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "M56_-tkKsZ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\\\nThis paper proposes a new 'baseline' for attribution methods tailored to deep neural networks. DNN attribution methods like integrated gradients, deep lift and others require a baseline to compare to as part of the computation. The choice of a baseline has been controversial in the literature, and a good method to select a baseline remains an open problem. This paper seeks to address that problem. Specifically, this paper seeks to develop a baseline for one-vs-one explanations as opposed to one-vs-all explanations. Consider an MNIST model, a one-vs-one attribution would attribute why an input is say a '2' and not a '4', i.e., it is contrastive against a particular target class and not all classes. This paper proposes to use a StarGAN for generating these baselines. The paper then evaluates explanations derived using the new baseline and shows that they explanations 'perform' better. \n\nOverall, I think the paper tackles an important problem, but I have several concerns with the motivation, the appropriateness of the baseline definition in this work, and the evaluation. I'll expand on these concerns in the later part of the review, so I am not recommending an accept in its current form. \n\nSignificance/Quality\\\nThe paper tackles an interesting and potentially challenging problem. However, motivation is still somewhat unclear, and there are critical problems with the evaluations used as justification here. I go into these at the end of this review. \n\nClarity/Writing\\\nThe paper is generally easy to follow. I problem I had reading it is that there are a few sentences that are stated as fact without any justification. For example, the paper notes, \"The minimum distance training sample in section 2.2 is a true class-targeted baseline proposed in the past.\" What is a 'true' class-targeted baseline? Such statements should probably be reformulated. \n\nMinor changes\nLasted paragraph of section 2.3: \"has posted a challenging\", posted is probably not the desired word here. \n\nQuestions and Concerns\n\n- Motivation: it is still not clear to me why a one-vs-one attribution is desirable? More should probably be done here to motivate this. The biggest need though is motivation for the one-vs-one attribution baseline. In several statements, the paper alludes to properties of baselines used in expected gradients and other methods, stating the reasons why these baselines are undesirable. I agree, but why should a one-vs-one baseline be preferable to these? Ideally, the paper will set out a list of desirable properties; then show that the baseline derived from GANMEX satisfies these. It is still not clear to me why a notion of minimum distance in a different target class is the right one. Can the authors say more about why this should be the case? \n\n- Evaluation: I'll preface my concerns here with the fact that I think evaluating model attributions or explanations in general is a difficult and open problem. This said, I don't think any of the evaluations presented in this paper can be taken as showing that the GANMEX baseline is the desirable one. First, the perturbation-based evaluation does not provide consistent rankings (see: Tomsett et. al. (AAAI Sanity checks for saliency metrics)). I suspect the gini index will have the same problems as those discussed in the Tomsett et. al. paper. The sanity checks themselves, i.e. the cascading randomization, will tell you if a method should be ruled out and not whether a method is effective. Consequently, I don't think the sanity checks can say much in judging baselines. Having said all of this, I think the way to evaluate a baseline is to take a task where the truth ground-truth rankings are known a priori, train a model to respect and align to the true ground truth. Now one can compare attributions from such a model for a normal baseline and a baseline from GANMEX. Assuming the attribution method itself is a reliable one, then one can quantify improvements due to the GANMEX baseline. \n\nA paper that might be related to this work that also incorporated generative modeling: https://arxiv.org/abs/1807.08024.pdf\n\nOverall, the concerns above make me hesitant about this current draft; however, I am happy to revise my assessment if the authors think I am wrong. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "02S6oHO7VfN", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper claims to present a novel GAN-based model explainability, for generating one-vs-one explanations, by incorporating to-be-explained classifier as part of the GAN. They use GANs to produce a baseline image which is a realistic instance from a target class that resembles the original instance.\n\nPositive aspects:\n- a novel approach for generating one-vs-one explanation baselines leveraging GANs\n- the proposed approach improves the saliency maps for binary classifiers \n\nNegative aspects:\n- the paper lacks clarity\n- the approach is demonstrated on cherry-picking examples. Have doubts of its generalization capability\n\nPlease find below some of my concerns:\n1. Your claim: \"...we use GANs to produce a baseline image which is a realistic instance from a target class that resembles the original instance\". Why do you need a GAN? Why don't you use a network to generate a confusion matrix to analyze the performance of the classifier? And based on this analysis you could explain why, for instance, the digit '0' is classified as a '6'. \n2. Related with the previous point, your analysis is very limited. You assume '0' is classified as '6'. Could '0' be classified as an '8' or '9'? It is not clear from your analysis. There are no comments on these cases. Looks like your examples to defend your approach are cherry-picked.\n3. I am not sure how to interpret Figure 2. \nSome other comments:\n1. The paper lacks novelty. The authors' contribution is not clear.\n2. The experimental validation is limited and not convincing. The authors use just some simple datasets (MNIST, SVHN). What about more complex datasets, like CIFAR10, LSUN, etc.? Could your approach explain the mis-classification in these cases?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work investigates the choice of a 'baseline' for attribution methods. Such a choice is important and can heavily influence the outcome of any analysis that involves attribution methods. The work proposes doing (1) one-vs-one attribution in a sort of contrastive fashion (2) generating baselines using StarGAN.

The reviewers have brought out a number of valid concerns about this work:

1. One-vs-one attribution appears to be novel, and distinctive enough from the more prevalent "one-vs-all" formulations. I am perhaps more optimistic than the reviewers that such a formulation is in fact useful, but I can see where the hesitancy can come from.
2. It's not clear that the evaluation shows that the proposed method is in fact superior to the others. All the reviewers touched upon this one way or another.
3. Somewhat simplistic datasets used for evaluation (noted that there are CIFAR10 results in the rebuttal).

This was more borderline than the scores would indicate. I thank the authors for the extensive replies and extra experiments. I encourage them to incorporate more of the feedback and resubmit to the next suitable conference. I do believe that doing experiments on ImagetNet (like previous work does, such as IG) would be quite worthwhile and convincing. I suspect the computational expense could be mitigated by re-using pretrained networks, of which there are many available for ImageNet specifically.
Decision: Reject