Saliency is a Possible Red Herring When Diagnosing Poor Generalization
Authors: Joseph D Viviano
Becks Simpson
Francis Dutil
Yoshua Bengio
Joseph Paul Cohen
Venue: ICLR 2021 Poster (2021)
Keywords: Feature Attribution
Generalization
Saliency

Abstract: Poor generalization is one symptom of models that learn to predict target variables using spuriously-correlated image features present only in the training distribution instead of the true image features that denote a class. It is often thought that this can be diagnosed visually using attribution (aka saliency) maps. We study if this assumption is correct. In some prediction tasks, such as for medical images, one may have some images with masks drawn by a human expert, indicating a region of the image containing relevant information to make the prediction. We study multiple methods that take advantage of such auxiliary labels, by training networks to ignore distracting features which may be found outside of the region of interest. This mask information is only used during training and has an impact on generalization accuracy depending on the severity of the shift between the training and test distributions. Surprisingly, while these methods improve generalization performance in the presence of a covariate shift, there is no strong correspondence between the correction of attribution towards the features a human expert have labelled as important and generalization performance. These results suggest that the root cause of poor generalization may not always be spatially defined, and raise questions about the utility of masks as 'attribution priors' as well as saliency maps for explainable predictions.

Published as a conference paper at ICLR 2021
SALIENCY IS A POSSIBLE RED HERRING WHEN DIAGNOSING POOR GENERALIZATION
Joseph D. Viviano 1.2,, 1.2., Becks Simpson', Francis Dutil?, Yoshua Bengio 13, & Joseph Paul Cohen
Milla Québec Artificial Intelligence Institute, Université de Montréal 2 IImagia Cyberneties
CIFAR Senior Fellow
josepheviviano
josephe josephpcchen.ccm hoseeppeeesspppphccccccee...co com
ABSTRACT
Poor generalization is one symptom of models that learn to predict target variables using muariously-oorreltted image features present only in the training distribution instead of the true image features that denote class. It is often thought that this can be diagnosed visually using attribution (aka saliency) maps. We study if this assumption is correct. In some prediction tasks, such as for medical images, one may have some images with masks drawn by a human expert, indicating a region of the image containing relevant information to make the prediction. We study multiple methods that take advantage of such auxiliary labels, by training net- works to ignore distracting features which may be found outside of the region of interest. This mask information is only used during training and has an impact on generalization accuracy depending on the severity of the shift between the training and test distributions. Surprisingly, while these methods improve generalization performance in the presence of a covariate shift, there is no strong correspondence between the correction of attribution towards the features a human expert has la- belled as important and generalization performance. These results suggest that the root cause of poor generalization may not always be spatially defined, and raise questions about the utility of masks as "attribution priors" as well as saliency maps for explainable predictions.
INTRODUCTION
A fundamental challenge when applying deep learning models stems from poor generalization due to covariate shift (Moreno- Torreoo-Torresaoe al., 2012) when the probably approximately correct (PAC) learning i.idd assumption is invalid (Valiant 1984) i.e. the training distribution is different from the test distribution. One explanation for this is shortcut learning or incorrect feature attribution, where the model during training overfits to a set of training-data specific decision rules to explain the training data instead of modelling the more general causative factors that generated the data (Goodfellow
et al. 2016 Reed & Marks
1999: Geirhos al. (2020) Hermann & Lampinen 2020 Parascandolo 2020 Arjovsky et al. 2019 Zhang et al. 2016).
ct al.
In medical imaging, poor generalization due to test-set distribution shifts are common and this prob- lem is exacerbated by small cohorts. Previous work has hypothesized that this poor generalization in part due to the presence of confounding variables in the training data such acquisition site or other image acquisition parameters because attribution maps (aka saliency maps; (Simonyan et al. (014 produced by the trained model do not highlight features that a human expert would use to make a diagnosis (Zech et al., 2018 DeGrave et al. 2020; [Bageley et al.) 2019 Zhao et al., 2019 Young et al.) (2019). Previous researchers have made the assumption that saliency maps can demonstrate that the model is not overfit or behaving unexpectedly (Pasa et al.) 2019 [schand! al. 2020). We started this work under the same assumption only to find the contradictory evi- dence we present in this paper. In this work, we set out to test the hypothesis that models with good generalization properties have attribution maps which only utilize the dlas-discriminative features to make predictions, by explicitly regularizing the models to ignore confounders using attribution priors (Erion et al.) 2019 Ross et al. 2017). i.e., to make predictions using the correct anatomy (as doctor would). We evaluated whether this regularization would A) improve out of distribution generalization, and B) change feature attribution to be more like the attribution priors. If there exists relationship between the attribution map and generalization performance, we would expect these

---

Published as a conference paper at ICLR 2021
regularizations to positively impact both generalization and attribution quality simultaneously. To evaluate attribution quality, we define good attribution to be an attribution map that agrees strongly with expert knowledge in the form of a binary mask on the input data.
We show that the existing and proposed ization in the presence of train-test distribution shift. However, while feature- iatpurr-aatribtion-wwaeaae methods change the attribution maps relative to baseline, there is no strong correlation between generalization performance and good attribution. This in turn challenges the assumption made in previous works that the "incorrect" attribution maps were indicative of poor generalization perfor- mance. This suggests that A) efforts validate model correctness using attribution maps may not be reliable, and B) that efforts to control feature attribution using masks on the imput may not function as expected. All code and datasets for this paper are publicly available Our contributions include:
methods help facilitate general-
A synthetic dataset that encourages models to overfit to an easy to represent confounder instead of a more complicated counting task.
Two new tasks constructed from open medical datasets which have a correlation between the pathology and either imaging site (site pathology correlation; SPC) or view (view pathology correlation; VPC), and manipulate the nature of this correlation differently in the training and test distributions to create a distribution shift (Figure 5 introducing confounding variables observed in previous work (Zhao et al.. 2019 DeGirave et al.) 2020).
Evaluation of existing methods for controlling feature attribution using mask information; right for the right reasons (RRR: Ross al. (2017)). GradMask (Simpson et al. 2019). and adver-
sarial domain invariance (Tzeng et al.) (0177GGann & Lempitsky (013)..
A new method for controlling feature attribution based on minimizing activation differences between masked and unmasked images (ActDiff))
Evaluate the relationship between generalization improvement and feature attribution in real- life out of distribution generalization tasks with traditional classifiers.
2
RELATED WORK
it well-documented phenomenon that convolutional neural networks (CNNs), instead of building object-level representations of the input data, tend to find convenient surface-level statistics in the training data that are predictive of class (Jo & Bengio 2017). Previous work has attempted to reduce the model's 's proclivity to use confounding features by randomly masking out regions of the input Vrees & Taylor 2017), forcing the network to learn representations that aren't dependent on single input feature. However, this regularization approach gives no control over the kinds of representations learned by the model, so we do not include it in our study.
Recently, multiple approaches have proposed to control feature representations by penalizing the model for producing saliency gradients outside of regions of interest indicating the class- discriminative feature (Simpson et al., 2019 Zhuang et al., 2019; Rieger et al., 2019). These ap- proaches were introduced by Right for the Right Reasons (RRR), which showed impressive im- provements in attribution correctness on synthetic data (Ross et al. 2017). The follow-up work has generally demonstrated a small improvement in generalization accuracy on real data, and much more impressive results on synthetic data. Another feature attribution
regularization approach additionally dealt with class imbalances by increasing the impact of the gradients inside the region of interest of the under-represennted class Zhuang et al. (2019)
One alternative to saliency-basee methods, which can be noisy due to the ReLU activations allowing irrelevant features to pass through the activation function (Kim et al., 2019), would be leverage methods that aim to produce domain invariant features in the latent space of the network. These methods regularize the network such that the latent representations of two or more "domains" are encouraged to be as similar as possible, often by minimizing a distance metric or by employing an adversary that is trained to distinguish between the different domains (Kouw & Loog 2019 Ganin & Lempitsky 2015: Tzeng et al. 2015 Liu & Tuzell 2016). In this work, we view the masked version of the input as the training domain and the unmasked version of the input as the test domain, and compare these approaches with saliency- based approaches for the task of reducing the model's reliance on confounding features. To the best of our knowledge, these strategies have not been tried to control feature attribution.
https:://// thub. wwtpwb.ww.//oeeeeeeeeeeeee httpb//.ww//ppsehh///iiiaa/////e sencslyl--eeeeeeee wed-reddeeer
2

---

Published as a conference paper at ICLR 2021
3 METHODS
Domain Invariance with the Activation Difference and Adversarial Loss: Leveraging ideas from domain adaptation, we introduce two methods designed to make the model invariant to features arising from outside of the attribution priors. The first is the embarrassingly simple Activation Difference (ActDiff) approach, which simply penalizes, for each input, the 1.2-normed distance between the masked and unmasked input's latent representations. This loss most similar a style transfer approach which transforms random noise image into one containing the same layer- wise activations as some target image (Gatys et al. (0155) to apply a visual aesthetic to some input semantic contents, and encourages the network to build features which appear inside the masked regions even though it always sees the full image during training. Therefore we minimize
Lact
Σ
Lats + >act )xmallo((Mmaeke)) (x)))22
(Xmarkedx))D
where (x) are the pre-activation outputs for layer I of the n-layer encoder f(x) when the network is presented with the original data x, 01(Xmasked)
T Fmakk
a
Emask:
the pre-activations outputs for layer I when presented with masked data Xmasked. and Lay is the standard cross entropy loss. We define Xmasked X- X-Xseg +-suffle(x) xeeg where background pixels are shuf- fled uniquely for each presentation to the net- work in order to destroy any spatial information available in those regions of the image without introducing any consistent artefacts into the im- age or shifts in the distribution of input intensi- ties.
dxt
Shared Weightt Weights
Saliency loss
OO Classification loss
Domain Invariance loss
Figure 1: Schematic of the model used in all ex- periments. The backbone is an 18-layer ResNet. The domain invariance penalties, ActDiff and Ad- versarial, are applied to the linear layer after aver- age pooling (although they could be applied at any location in the network). In contrast, the saliency penalties, GradMask and Right for the Right Rea- sons, applied on the input space. All losses denoted using standard dashed lines.
This method is at high level similar to using maximum mean discrepancy (MMD) for do- al.) 2016))
main adaptation (Baktashmotlagh 2013) where instead of minimizing the distance between the means of the two domain distri- butions, we instead minimize the distance be- tween the features directly. One could replace the L2 norm with any Lk norm, and choices of might be useful when regularizing larger latent representations as 7.2 distances collapse to a constant value in extremely high dimensional spaces due to the curse of dimensionality (Aggarwal et al. 200I). We found during experimentation that regularizing the pre-activations led to better results at the cost of longer time to convergence, perhaps because the L2 norm is more effective when the feature vectors are not sparse, but we leave this conjecture future work.
The second approach we explore employs a discriminator D(-) optimized to distinguish between latent representations arising from passing Xmasked or through the encoder f(.) (Tzeng et al. 2017 Goodfellow et al., 2014). Simultaneously, we optimize the encoder ʃ(.) to fool the discrim- inator and still produce representations that are predictive of the output class. Therefore, D and are optimized using the Lo and L, respectively:
LD = Addiec
(ogD (Xmmsske())))) + E Exxasnee [lgg(1 - -D(((x))))
(2)
=y= Lelf + Adise (Ex(log(( (xf((Xaaskee ] + Ex (f)))(((x)))
(3)
This approach similar to the one employed in (Janizek al. 2020), where the authors encouraged the model to be invariant to the view of the X-Ray (Lateral vs PA), and here "view" denotes whether the image is masked or not. ((.) had three fully-connected hidden layers size 1024 before out- putting binary prediction. To facilitate the stability of training, we updated D(-)) more frequently than the encoder treating this ratio as a hyperparameter. We optimized D(.) independently using Adam with distinct learning rate, and applied spectral normalization the hidden layers of D(-).
Direct Attribution Control with the Right for the Right Reasons and GradMask Loss: These input gradient attribution regularizers directly control which regions of the input are desirable for
3

---

Published as a conference paper at ICLR 2021
determining the class label by penalizing saliency outside of defined input mask. The most basic gradient based "input feature attribution" can be calculated as 2 for each input x (Simonyan an al, 2014 Lo et al., 2015). In the binary classification case, RRR (Ross et al.) (017) calculates saliency of the summed log probabilities of the output classes with respect the input
Lats + Aeer à (log(p)) + +lggpi)))
Fer
Σ (Xx
1 (1-Xxs (sexs
(4)
0x
where (1 - X is a binary mask that covers everything outside the defined regions of interest.
The numerator in the RRR loss can be extended to the multi-class case by substituting Σt log(pk))
GradMask (Simpson et al. 2019) similarly calculates saliency using the contrast between the two
output logits, and is only defined for the binary classification case,
graddmask
Σ
Lat + Agradmask
i 100
01
(1 -xeeg)
(5)
dx
X)ED
where do and أ1 are the predicted outputs for our two classes.
Classify Masked: We evaluated the effect imposing the attribution prior by simply training a model using masked data (and evaluating it using unmasked data) as a control experiment. The data was masked by shuffling the pixels outside of the mask during training, as was done for the domain invariance experiments to produce Xmasked. We refer to these experiments as Masked.
4 EXPERIMENTS
Feature Attribution Analysis and Visualization: We evaluated our ability to control feature attri- bution using three methods. We first calculated the input gradient the absolute gradient G of the input with respect to the prediction made for all images of the positive class G cct al.] 2014: [Ancona et al., 2018). We also calculated integrated gradients, which produces attribu- tion maps that are invariant to the specific model trained (or function f()) and are sensitive to all input features that drive the prediction (Sundararajan et al. 2017). This is done by integrating the gradients along the straightline path between the input image n, and all-zero baseline image a/,
(Simonyan
dx
1G, := (z: - 2 x Ju x(=='ax(x==x)) da. The integral was approximated over 200 steps for each image using GaussLegendre quadrature. We finally evaluated a non gradient-based attribution method occlusion where we divided the input image into a 15 x 15 grid, and for each element of the grid, recorded the magnitude of the change to the model's output logits if that region is removed from the input (by setting these regions zero) (Zeiler & Fergus 2014). For the two gradient- attribution methods, we smoothed the resulting saliency map with a small Gaussian kernel (a 1) to remove pixel-wise variance in the attribution values that amount to noise. The attribution maps were then carried forward to calculate binarized version such that the top P percentile attribution values were set to 1, where p is dynamically set to the number of pixels in the mask that is being compared to. These binarized values were used to calculate the localization accuracy between bi- narized attribution map and the ground-trume truth segmentation using the intersection over union (IoU; B) A) ) for all images in the test sets for all models trained. To aid in visualization, also thresholded these attribution maps arbitrarily at the 50th percentile so that the most attributed regions of the image are easily seen when overlaid on the anatomy. We present visualizations of the results for all three methods in the Appendix. These methods were implemented using the Captum library (Kokhlikyan et al.) 2020).
des
Model, Optimization, Hy Hyperparameters, and Search Parameters: The backbone of all exper- iments was the 18 model from Torchvision (He et al. [016]] Marcel & Rodriguez] 2010). For medical dataloaders we use the TorchXRay Vision library (Cohen et al.J [020b]aa] a). To see an overview of how the losses described above relate to the architecture, see Figure ! All models were trained using the Adam optimizer (Kingma & Ba 2014) using early stopping on the validation loss.
Hyperparamters were selected for all models using Bayesian hyperparameter search with single seed, 5 random initializations, 20 iterations, and trained for a maximum of 100 epochs with a pa- tience of 20. Final models were trained using the best hyperparameters found for 100 epochs on 10 seeds. The learning rate for all models was searched on a log uniform scale between [10-5 10 -0- The ActDiff, Adversarial, GradMask, and RRR lambdas were all searched between [10-4 10), each
4

---

Published as a conference paper at ICLR 2021
Train Samples
Valid Samples
Class 0
Class 1
Class 0
Class
Figure 2: Example images from Dirain and Dealid from both classes. In both distributions, cross size can vary between samples. In Dtrain. two crosses (denoting class 0) are always accompanied by a box ze in the bottom right-hand corner, while a single cross (denoting class 1) always accom- panied by confounder in the bottom left- left-hand corner. In Dvalid, the relationship between classes and crosses remains the same, but the logic governing the location of the counfounder is reversed. The confounder is indicated with red arrow.
on a log uniform scale. For Adversarial, we searched for the optimal discriminator encoder train- ing ratio of [2 10 1], and the discriminator learning rate was searched on a log uniform scale between [10-1 10-2). To retain context around the masked region in the Synthetic dataset, we di- lated the mask by applying Gaussian blur with hyperparameter to the mask and then binarize the result; a was searched on a uniform scale between [o 2]. For all experiments, the batch size In performing our hyperparameter searches, we found that certain algorithms were much easier to tune than others. We present the final best validation AUCs encountered for all iterations for all experiments in Appendix Figure A.1 and Table 3
was 16, weight of the classification cross entropy loss
4.1 SYNTHETIC DATASET
Consider a classification problem where there exists some confounder feature x, in the data (a vector of variables a:) that is perfectly correlated with one of the output classes in the training distribution Drrain such that ply 1|x,) 1, while in the validation distribution Doatid- p(y 0/xze) (Figure 2). In this scenario, predicting using ze is casier than predicting using the true features that would denote class membership and a classifier trained on Dirain with traditional classification loss would predict the incorrect class with 100% probability on Deatid-
Protocol:
We generated a dataset with these conditions where class membership is denoted by the number of crosses present in the image (1 vs. 2 crosses) that appear any position, but there exists in all Derain images a confounder that always appears in the same position and is perfectly correlated with y (Figure The logic governing the position of the confounder was inverted for Dratid- This dataset had 500 training, 128 validation, and 128 test examples respectively. Segmentations were generated for all crosses in the dataset facilitate all feature-attribution controlling losses. In cases where the model relies on the confounder to make a class prediction, we expected 0.0 AUC for the validation and test sets.
Synthetic
The mean and standard deviation of the validation AUC over all 10 models for the 100 epochs of training for the best-performing hyperparameters are shown in Figure 3 show- ing the variance in performance over model ini- tializations and data splits. We demonstrate the effect of controlling feature attribution by showing the mean integrated gradient map cal- culated for 500 positive examples in the test in Figure
Results:
Experiment
0.8
Valld AUC
ActDlff
Masked
0.4
Adversarial
0.
GradWWak
40
60
100
Epoch
Figure 3: Synthetic Dataset Valid AUC over 100 epochs, averaged over 10 seeds.
The validation curves (Figure 3) demonstrate that the baseline model quickly lears the shortcut in the training set and shows below-chance generalization (AUC=0). The Masked approach is never given the opportunity build representation of the confounder and is therefore able general- ize while still giving attribution to the confounder. The domain invariance approaches, ActDiff and Adversarial also learn the task, although convergence is much more difficult for the Adversarial ap- proach. The two saliency-baue based training and never reach optimal performance, although they do prevent the model from overfitting to the confounder. The saliency maps for these models (Figure 4 show that all feature attribution controlling approaches (C-F) successfully refocus the model's s attention away from the confounder, unlike the baseline and Masked models (A-B), leading improved generalization. This strong lationship between IoU and AUC for this dataset can be observed for all seeds in Figure g As we will see, this relationship does not hold for real world data.
regularizers, GradMask, and RRR, show unstable
5

---

Published as a conference paper at ICLR 2021
Baselline
Marked Adverssrial
D: ActD if
Mrac
RRR
Syathetie lG
Figure 4: The average integrated gradients (IG) saliency maps from 500 randomly-selected Dteat images for the Synthetic dataset. The first image shows the mean of the samples, with the mean of the masks superimposed red. The remainder of the panels show the mean of the gradients after taking the absolute value, smoothing, and thresholding at the 50th percentile.
4.2 X-RAY DATASET WITH
CORRELATION
Protocol: We introduced a covariate-shift into a joint dataset X-Rays drawn from two different imaging centres: the PadChest (Bustos et al. 2019) sample and the NIH Chestx-Ray8 (Wang et al.) 2017] sample. In Figure 5 we can see examples of the distribution shift between imaging site (top) and imaging view (bottom; used in the following section). In the case that the prevalence of disease is any way correlated with either of these confounders, the model is likely to learn to use these differences to make prediction regardless of their medical relevance (rightmost column). In this dataset, models were required to predict the presence of Cardiomegaly (enlarged heart). A site-driven overfitting signal has previously been reported when combining these datasets (Zech eet al. 2018), where the model often attributes importance to the patient's shoulder when making prediction. We also observed site-driven differences in regions far from the lungs (see the mean X-Ray from each dataset in Figure 5p. and therefore hypothesized we could improve overfitting performance by masking out the edges of the image using a circular mask. To induce the covariate shift, we preferentially sampled the positive class from one of the two imaging centres in the training set to produce a site-pu tath-olly correlation (SPC), and then produce validation and test set where the reverse relationship is true.
In the training set, 90% of the unhealthy pa- tients were drawn from the PadChest dataset and the remaining 10% of the unhealthy pa- tients were drawn from the NIH dataset, and the reverse logic was followed for the validation and test sets. In all splits the classes and site distributions were always balanced, making it tempting for the classifier to site-specific feature when predicting the class in the pres- ence of site-pathology correlation (Nerain 5542, Nealdd = 2770, Ntest = 2770). We also constructed a version of this dataset where the classes are drawn equally from both sites for the training, validation, and test sets, i.e., dataset where there exists no site pathology correlation (No SPC). All images were 224 x 224 pixels.
mcan//ili
mcan/PC)
Diffrrnnce (no SPC)
Offference (SPC)
moan/PAI
mcan/AP) (AP)
Difference (nc VPC)
Offference (PC)
Figure 5: Illustration of the effect of Site- Pathology Correlations (SPC; top row) and View- Pathology Correlations (VPC; bottom row). The first two columns contain the mean of all X- Rays collected from a particular site/view. The third column shows the class-conditional mean difference between images taken from sick and healthy patients when pathology is uncorrelated with site/view, and the fourth column shows the same when pathology is correlated with site/view.
Results: We trained each model on both the SPC (90%) and No SPC datasets (Table [ A ResNet trained on the No SPC dataset scores test AUC of 0.93 + 0.01, while one trained in the presence of strong SPC scores test AUC of 0.70 + 0.05, indicating poorer generalization under SPC. Validation curves across seeds show that model optimization was much easier and more consistent for all models when trained on the No SPC dataset (Appendix Figure A.2p. The AcIDiff, Masked, and GradMask approaches only im- proved classification performance over the baseline model in the presence of an SPC (GradMask performed only slightly better than baseline for the No SPC dataset). For ActDiff and GradMask, an improvement classification performance was associated with improvement in attribution (mea- sured as an improvement in the loU between the binarized saliency map and the masks), but this pattern was not observed for the masked experiment, even though the Masked experiment produced superior generalization (it is worth noting that the Masked performance was more variable across seeds). Furthermore, the GradMask approach improved attribution in the No SPC case, but this was not associated with meaningful improvement in generalization performance. Figure 6 shows the mean saliency map calculated from a subset of all test images from all 10 models. Differences the baseline model's attribution (A) in the SPC and No SPC case are apparent: in particular, the SPC

---

Published as a conference paper at ICLR 2021
Basellne
Markedd
Adierrsriill
D: ActD if
Grac
ARR
SPC,
Image
No SPC. iG
Figure 6: Mean integrated gradients (IG) saliency maps from 500 randomly-selected Drest images in the X-Ray dataset with a site-pathology correlation (SPC; top), and without (No SPC; bottom) after taking the absolute value, smoothing, and thresholding at the 50 percentile across all 10 seeds. The first image shows the mean input image of the model over all examples shown, with the mask overlaid in red.
With SPC
Experiment
AUC
IoU Input Grad
ToU Integrated
ToU Occlude
Classification
0.70 1: 0.05
0.40 0.03
0.33 = 0.03
0.41 1 0.04
Baseline
Masked
0.77 +: 0.09
0.38 0.04
0.38 = 0.05
= 0.04
ActDiff
0.73 + 0.03
0.62 0.04
0.58 = 0.04
0.62 0.02
Domain Invariance
Adversarial
0.68 0.07
0.36
0.32 0.02
0.04
GradMask
0.74 0.03
0.57 0.02
0.50 = 0.04
0.50 0.04
Saliency Based
RRR
0.57 0.09
0.38
0.33 0.03
0.04
No SPC
Experiment
ACC
ToU Input Grad
ToU Integrated
ToU Occlude
Classification
0.93 1: 0.01
0.48 0.03
0.42 1: 0.03
0.49 1 0.02
Baseline
Masked
0.90 +: 0.01
0.42 0.02
0.39 0.03
= 0.03
ActDiff'
0.63 1: 0.20
0:21 0.25
0.26 0.24
0.21 1: 0.26
Domain Invariance
Adversarial
0.92 - 0.01
0.44
0.40 0.04
0.03
GradMask
0.94 0.00
0.55
0.02
0.48 = 0.03
0.53 = 0.02
Saliency Based
RRR
0.93
0.46
0.42 0.04
0.48 0.04
X-Ray Cardiomegaly test results for the best valid epoch over 100 epochs on the X-Ray dataset with and without a site-pathology correlation (SPC). Results averaged over 10 seeds with the standard deviation. Bold indicates better than Classification baseline and chance. IoU scores were calculated on 100 randomly-selected test-set images for each seed (total of 1000).
Table 1:
model shows greater reliance the shoulder for producing a prediction. The Masked experiment (B) exacerbates this behavior but accompanied by improved generalization with higher variance across sceds. In contrast, ActDiff and GradMask appear refocus the saliency away from the shoul- der (D-E), reflected in the increased loU scores for both the ActDiff and GradMask approaches for the SPC data. The weak relationship between IoU and AUC can be observed for all seeds in Figure g These results provide evidence that controlling feature attribution only facilitates generalization in the presence of covariate shift, but improves saliency in both the presence or absence of covariate shift, implying a weak relationship between good attribution and good generalization.
4.3 X-RAY DATASET WITH VIEW-PA NEW-PATHOLOGY CORRELATION
Protocol: Here made use of the RSNA Pneumonia challenge (Shih al. 2019) dataset, which includes images taken from antero-posterior (AP) and posterior-anterior (PA) views, and where the task is predict pneumonia. We introduced covariate shift into this dataset by sampling the positive and negative classes during training such that 90% of the positive classes are sampled from one view in the training set and the opposite view in the validation and test sets. We call this view- pathology correlation (VPC). After drawing the samples from both views and balancing the classes, the data in each split was as follows: Nirain = 2696, Nyatid = 1348, Nrast = 1348. All images more bounding boxes indicating the predictive regions. In Figure we
with Pneumonia have can see the strong effect the VPC, such that the lungs, shoulders, and torso show strong interclass differences in the VPC case. We also constructed No VPC dataset in the same way during the previous experiments, and all images were 224 x 224 pixels.
Results: Similarly the SPC case, baseline AUC in the presence of VPC (0.20 + 0.03) is substan- tially lower than the No VPC baseline (0.76 + 0.02), and additionally far below chance performance (0.50), illustrating the strong covariate shift in this dataset. Appendix Figure Ah2lshows that valida-

---

Published as a conference paper at ICLR 2021
Basellie
Marked
Adierssriill
ActD if
Mrac
RAR
With VNC
Image
No VPr.
Figure 7: Mean integrated gradients (IG) saliency maps from 500 randomly-selected Drest images in the RSNA dataset with a view-pathology correlation (VPC; top), and without (No VPC; bottom) after taking the absolute value, smoothing, and thresholding at the 50 percentile across all 10 seeds. The first image shows the mean input image of the model over all examples shown, with the mask overlaid in red.
With VPC
Experiment
AUC
IoU Input Grad
IoU Integrated
IoU Occlude
Classification
0.20 = 0.03
0.10 0.02
0.05 = 0.02
0.08 0.02
Baseline
Masked
0.56 0.16
0.07 0.03
0.08 + 0.02
0.04 0.01
ActDfff
0.68 0.05
0.08
0.02 + 0.00
0.01 0.07 1 0.04
Domain Invariance
Adversarial
0.62 1 0.10
0.10 1.00100 0.05
0.05 1 0.03
GradMask
0.49 = 0.06
0.17 0.06
0.08 0.02
0.08 = 0.03
Saliency Based
RRR
0.21 1 0.04
0.11 0.02
0.05 1: 0.01
0.08 1 0.02
No VPC
Experiment
AUC
IoU Imput Grad
Integrated
Occlude
Classification
0.76 = 0.02
0.17 0.02
0.10 + 0.02
0.11 = 0.01 0.04 0.02
Baseline
Masked
0.50 0.02
0.07
0.08 + 0.02
ActDfff
0.60 0.02
0.13 0.01
0.05 + 0.01
0.01
Domain Invariance
Adversarial
0.63 1 0.02
0.09 0.02
0.04 i 0.01
0.07 0.01
GradMask
0.75 0.01
0.16 0.03
0.11 + 0.02
0.11 0.01
Saliency Based
RRR
0.73 + 0.01
0.16 + 0.04
0.10 + 0.02
0.12 + 0.02
RSNA Pneumonia test results for the best valid epoch over 100 epochs the RSNA dataset with and without a view-pathology correlation (VPC). Results averaged over 10 seeds with the standard deviation. Bold indicates better than both the Classification baseline and chance, IoU scores were calculated on 100 randomly-selected test-set images for each seed (total of 1000).
Table 2:
tion performance was more variable in the presence of VPC between models, and that performance was generally lower. The ActDiff. Masked, and Adversarial approaches improved classification per- formance over the baseline model and chance performance in the presence of VPC. For only the GradMask model was an improvement in classification performance associated with an improve- ment in saliency, and the Masked experiment produced IoU scores substantially worse than baseline (again, Masked performance was more variable across seeds). No model improved attribution or generalization performance over baseline in the No VPC case. Figure 7 shows the mean saliency map calculated from a subset of all test images from all 10 models. Differences in the Baseline model's saliency (A) in the VPC and No VPC case apparent: in particular, the VPC model shows a greater reliance on the abdomen and shoulders for producing a prediction. The Masked experiment (B) again exacerbates this behavior but is accompanied by improved generalization baseline. The Adversarial, ActDiff. and GradMask models (C-E) appear to refocus the attribution toward the lungs for for the VPC case, but the mean Adversarial IoU score dropped over the full dataset, and ActDiff' resulted in minor loU score increase. The GradMask approach produced the best attribution as measured by IoU but does not produce model that outperforms random guessing on the test set. Again, these results suggest that the use of attribution priors can have a substantial effect on generalization performance in the absence of meaningful changes to the saliency towards what would be expected from the attribution priors, and vice versa. The weak relationship between loU and AUC can be observed for all seeds in Figure
LIMITATIONS AND CONCLUSIONS
Limitations: Our results show that models explicitly regularized to not use features constructed from outside of the mask still attribute features outside of the mask test time. This is likely due
8

---

Published as a conference paper at ICLR 2021
the fact that convolutional models are still able to construct discriminative filters from inside of the mask which might share properties with features found outside of the mask. The model is free to use any aspect of the input image to make a prediction during test time. In particular, the methods explored are not guaranteed to negate any confounding variables that exist within the mask, which could explain these results, and might additionally give rise to underspecified trained models, as evidenced by the models with high test set variability across seeds (D' A'mour et al., (020). This is particularly true for the Masked models, for which the representations constructed from inside of the masks were not regularized in any way. Methods that address this underspecification would make for valuable future work. Furthermore, while the results suggest that attribution maps are a mislead- ing indicator of generalization performance, we only evaluated these algorithms using small sample sizes with large covariate shifts between the training and test distributions in a medical context. Fu- ture work should more thoroughly evaluate the relationship between generalization performance and feature attribution in other settings before we can draw strong conclusions about the non non-cxistence of this relationship. Furthermore, methods which control feature attribution without using masks for enforeing domain knowledge might show a stronger relationship between IoU and AUC, and future work should explore this possibility. These methods would also be more general, as input masks are only available in specific domains and can be expensive to obtain.
Conclusions:
We hypothesized that poor generalization performance could be partially attributable to classifiers exploiting spatially-distinct confounding features (or shortcuts) as have been previously diagnosed using saliency maps (Badgeley
al. 2019 Zech et al. 2018). We evaluated the perfor- mance of multiple mitigation strategies on a synthetic dataset and two real-world datasets exhibiting covariate shift, using attribution priors in the form of a segmentation of the discriminative features.
In our synthetic dataset, we defined a spatially-distinet confounder that prevented generalization for standard classifiers, and demonstrated that imposing an attribution prior would facilitate generalization and im- prove attribution maps. In real data, we found that attribution priors also facilitate generalization when there exists a covariate shift between the training and testing distri- bution, but these methods hurt generaliza- tion when there was not covariate shift. Surprisingly, improvement in generalization performance was not reliably accompanied by improvement in attribution: this is partic- ularly apparent for the AcIDiff model on the RSNA VPC dataset, which had meaning- ful positive impact on AUC and very nega- tive impact on attribution. Figure 8 summa- rizes the relationship between generalization (AUC) and feature attribution (IoU) across all seeds for all experiments with a covariate shift: the positive correlation between AUC and IoU apparent for the synthetic data was not present in the real world datasets.
Classification Performance vs. Localization Performance (Integrated)
0.7
0.6
Test loU
0.3
0.2
0.1
0.0
0.0
0.2
0.4
0.6
0.8
100
Dataset
Test
Approach
Synth
Baseline
XRay SPC
Domain Invariance
RSNA
Mask
Saliency
Figure 8: Test AUC vs. Test IoU (good saliency) for all seeds evaluated with covariate shift. Models presented grouped by "Approaches": baseline, input masking, domain-invann insaiiinivarianee, and saliency-based ap- proaches. All localizations were computed using in- tegrated gradients, see Appendix Figure A.13 for all attribution methods.
In summary, we find a tenuous relationship between good saliency and generalization performance in real datasets. Many models that exhibit good generalization performance do not obtain good at- tribution, and vice-versa. While our methods exert influence on the features the model constructs from the data, real data we found no evidence of a relationship between improved generalization and improved feature attribution. We now doubt the validity of using saliency maps for diagnosing whether model is overfit because improving generalization using attribution priors is not accom- panied by improved attribution.
9

---

Published as a conference paper at ICLR 2021
REFERENCES
Charu C Aggarwal, Alexander Hinneburg, and Daniel A Keim. On the surprising behavior of distance metries in high dimensional space. In International conference on database theory. Springer, 2001.
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. 2018 In International Conference on Learning Representations.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv: 09.999999, 2019.
Mareus Badgeley, John R Zech, Luke Oakden-Rayner, Benjamin Glicksberg. Manway Liu, William Gale, Michael V McConnell, Bethany Percha, Thomas M Snyder, and Joel T Dudley. Deep learning predicts hip fracture using confounding patient and healthcare variables. npj Digital Medicine, 2019.
Mahsa Baktashmotlagh,, Mehrtash T Harandi, Brian C Lovell, and Mathieu Salzmann. Unsupervised domain adaptation by domain invariant projection. In Proceedings of the IEEE International Conference on Computer Vision, 2013.
Mahsa Bakrashmotlagh, Mehrtash Harandi, and Mathieu Salzmann. Distribution-matching embed- ding for visual domain adaptation. The Journal of Machine Learning Research, 2016.
Aurelia Bustos, Antonio Pertusa, Jose-Maria Salinas, and Maria de Iglesia-Vayd, PadChest: A large chest x-ray image dataset with multi-label annotated reports. arXiv e-prints, 2019 art. 1XXiv11011.0741..
Joseph Paul Cohen, Mohammad Hashir, Rupert Brooks, and Hadrien Bertrand. On the limits of cross-domain generalization in automated x-ray prediction. In Medical Imaging with Deep Learn- ing. 2020a.
Joseph Paul Cohen, Joseph Viviano, Mohammad Hashir, and Hadrien Bertrand. TorchXRay Vision: 2020b.
A library of chest X-ray datasets and models.
Alexander D' D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beu- tel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew Hoffman, et al. Under- specification presents challenges for credibility in modern machine learning. arXiv preprint arXiv:2011.03335. 2020.
Alex J DeGrave, Joseph D Janizek, and Su-In Lee. AI for radiographic COVID-19 detection selects shortcuts over signal. 2020 medRxiv. doi: 10. 11111002200...13.2119999999
Terrance DeVries and Graham W. Taylor. Improved Regularization of Convolutional Neural Net- works with Cutout. 2017 1rXiv:/708.044
Gabriel Erion, Joseph D Janizek, Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Learning ex- plainable models using attribution priors. arXiv: 10670, 2019.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning, 2015.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. A neural algorithm of artistic style. arXiv:1508.0653 06.5088.0677.. 2015.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. arXiv:2004.07780, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, 2014.
10

---

Published as a conference paper at ICLR 2021
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Conference on Computer Vision and Pattern Recognition, 2016. doi: 10.1 109/ CVPR.2016.90.
Katherine I. Hermann and Andrew K Lampinen. What shapes feature representations? exploring datasets, architectures, and training. arXiv:2006.1244 2020.
Joseph Janizek, Gabriel Erion, Alex J DeGrave, and Su-In Lee. An adversarial approach for the robust classification of pneumonia from chest radiographs. In Proceedings of the ACM Conference on Health, Inference, and Learning, 2020.
Jason Jo and Yoshua Bengio. Measuring the tendency of cnns to learn surface statistical regularities. CoRR, 2017.
Beomsu Kim, Junghoon Seo, SeungHyun Jeon, Jamyoung Koo, Jeongyeol Choe, and Taegyun Jeon. Why are saliency maps noisy? cause of and solution to noisy saliency maps. 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014.
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang. Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina. Carlos Araya, Siqi Yan, and Orion Reblitz- Richardson. Captum: unified and generic model interpretability library for pytorch, 2020.
Wouter Mareo Kouw and Marco Loog. A review of domain adaptation without target labels. IEEE fransactions on pattern analysis and machine intelligence, 2019.
Ming-Yi Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. In Advances in neural information processing systems, 2016.
Henry Z. Lo. Joseph Paul Cohen, and Wei Ding. Prediction gradients for feature extraction and analysis from convolutional neural networks. 2015 In International Conference on Automatic Face and Gesture Recognition. doi: 10.1109/F.221157766115..
Sébastien Marcel and Yann Rodriguez. Torchvision the machine- vision package of torch. In Pro- ceedings of the 18th ACM international conference on Multimedia, 2010.
Jose G. Moreno- Torres, Troy Raeder, Roeío Alaiz-Rodríguez, Nitesh V. Chawla, and Francisco Herrera. A unifying view on dataset shift in classification. 2012 Pattern Recognition. doi: PA.016/..AATOOG..200 1.0000019.
Giambattista Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard vary. MXXi:2009.003399 2020.
FF Pasa, V Golkov, F Pfeiffer, D Cremers, and D Pfeiffer. Efficient Deep Network Architectures for Fast Chest X-Ray Tuberculosis Screening and Visualization. Scientific Reports, 2019. doi:
11110884415988-01994555544444
Russell D. Reed and Robert J. Marks. Neural smithing: supervised learning in feedoorwadd artificial neural networks. MIT Press, 1999.
Laura Rieger, Chandan Singh, W James Murdoch, and Bin Yu. Interpretations are useful: penalizing explanations to align neural networks with prior knowledge. 1rXiv:1909.13588 2019.
Andrew Ross, Michael C Hughes, and Finale Doshi Velez. Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations. In International Joint Conference on Artificial Intelligence, 2017.
11

---

Published as a conference paper at ICLR 2021
George Shih, Carol C. Wu, Safwan S. Halabi, Mare D. Kohli, Luciano M. Prevedello, Tessa S. Cook, Arjun Sharma, Judith K. Amorosa, Veronica Arteaga, Maya Sulperin-Aizenlon Ritu R. Gill, Myma C.B. Godoy, Stephen Hobbs, Jean Jeudy, Archana Laroia, Palmi N. Shah, Dharshan Vummidi, Kavitha Yaddanapudi, and Anouk Stein. Augmenting the National Institutes of Health Chest Radiograph Dataset with Expert Annotations of Possible Pneumonia. 2019 Radiology:
Artificial Intelligence. doi: 10.1 148/114ai 2019180041.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. In International Conference on Learning Representations (ICLR), 2014.
Becks Simpson, Francis Dutil, Yoshua Bengio, and Joseph Paul Cohen. GradMask: Reduce Over- fitting by Regularizing Saliency. In Medical Imaging with Deep Leurning Workshop, 2019.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. arXiv preprint 1703:007703.1366 2017.
Philipp Tschandl, Christoph Rinner, Zoe Apalla, Giuseppe Argenziano, Noel Codella, Allan
Halpem,, Monika Janda, Aimilios Lallas, Caterina Longo, Josep Malvehy, el. Human-computer collaboration for skin cancer recognition. Nature Medicine, 2020.
Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultaneous deep transfer across domains and tasks. In Proceedings the IEEE International Conference on Computer Vision, 2015.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, 2017.
L. G. Valiant. A theory of the learnable. In Proceedings of the Annual ACM Symposium on Theory of Computing. 1984 Association for Computing Machinery. doi: 10.1 10.1145/800057..088 8007000
Xiaosong Wang. Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Sum- mers. Chestx-ray8: Hospital-scale chest x-ray database and henchmarks on weakly-superrised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Kyle Young, Gareth Booth, Becks Simpson, Reuben Dutton, and Sally Shrapnel. Deep neural network dermatologist? arXiv: /2, 2019.
John R. Zech, Mareus A. Badgeley, Manway Liu, Anthony B. Costa, Joseph Titano, and Eric K. Oermann. Confounding variables can degrade generalization performance radiological deep learning models. CoRR, 2018.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision. Springer, 2014.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv:1611.03533 2016.
Qingyu Zhao, Ehsan Adeli, Adolf Pfefferbaum, Edith V Sullivan, and Kilian M Pohl. Confounder- Aware Visualization of ConvNets. 19Xiv: 2019.
Jiaxin Zhuang, Jiabin Cai, Ruixuan Wang. Jianguo Zhang, and Weishi Zheng. {CARE): Class attention to regions of lesion for classification on imbalanced data. In International Conference on Medical Imaging with Deep Learning - Full Paper Track, 2019 London, United Kingdom.
ACKNOWLEDGMENTS
We thank Julia Vetter and Pascal Vincent for insightful discussions. This work is partially funded by grant from the Fonds de Recherche en Santé du Québec and the Institut de valorisation des données (IVADO). This work utilized the supercomputing facilities managed by Compute Canada and Calcul Quebec. We thank PrcademicTorrente com for making data available for our research.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 ARCHITECTURE THE RESNET-18 MODEL
The backbone of the model used for all experiments was the 18-layer resnet available from torchvi- sion (Marcel & Rodrigue7, 2010) with small architecture adjustment the input of the network. The official model's 's input convolutional layer had a kernel size of 7, stride of 2, and padding of 3. In our experiments,
found this larger kernel size the inputs led less specific saliency maps (data not shown), and we therefore changed the kernel size to 3, stride to 1, and padding to for this layer. Immediately following this layer, the official model employed maxpooling operation. We found this operation to produce unusable noisy saliency maps, we also disabled this operation for all experiments.
A.2 HYPERPARAMETER SENSITIVITY
XRay SPC
XRay
Unregularized
Unregularized
Actdiff
Actdid/
Classify Masked
Classify Masked
Discrine ator
Discriminator
Gradmask
Gradmask
RAR
RRR
0.5
0.7
0.8
0.6
0.7
088
0.
AUI
AUC
RSNA VPC
RSNA No VPC
Unr
Unregularized
Actdiff
Actdiff
Classity Masked
Classity
Discrimingtor
Discriminaj
Grad
Gradmask
RAR
0.2
0.
0.4
0.6
08
0.0
0.5
0.60
0.5
0.70
0.75
AUC
AUC
Synthetic
Uniggularized
Actdiff
Classify Masked
Discriminator
Gradmask
RRR
0.2
0.4
0.6
0.8
AUC
Figure A.1: The best validation AUC for all hyperparameter settings tested during the model tuning. On the synthetic dataset, domain adapttaioon approaches were able to solve the task, as did the model that simply observed masked versions of the data. The saliency-base approaches were generally more sensitive to hyperparameters. In the presence of a covariate shift (SPC/ VPC), hyperparameter tuning was more difficult on all real-world data.
13

---

Published as a conference paper at ICLR 2021
A.3
VALIDATION CURVES FOR THE REAL WORLD DATASETS
XRay SPC
Xñay No SPC
Experiment
0.8
Beseline
Valid AUC
Valid AUC
0.6
0.6
Masked
0.4
0.4
Adversarial
GradMask
02
0.
RAR
RSNA VPC
RSNA No VPC
1.0
1.0
0.8
0.8
Valid AUC
Valid AUC
0.6
0.6
Epoch
Epoch
Figure A.2: All real dataset RX-Ray and RSNA) valid AUC across the 100 epochs of training. Mean and standard deviation presented over 10 seeds.
A.4
BEST HYPERPARAMETERS FOR ALL EXPERIMENTS
Dataset
Experiment
Learning Rate
Lambda
Disc Iteration Ratio
Disc Leamning Rate
Synthetic
Classificution
3.11 x 10-4
Masked
3 10-5
Actdiff
3.11 10-4
1.15 x 10-1
Adversarial
3.71 10-4
10:1
0.01
GradMask
10-2
2.03 10-1
RRR
10
7.56
X-Ray SPC
Classification
9.99 10-3
Masked
10-2
Actdiff
10-4
10-1
Adversarial
x
1 10
GradMask
10
3.85 10-1
RRR
7.98 10-4
x 10
X-Ray No SPC
Classification
10
Masked
6.87 10-4
Actdiff
10
5.78
Adversarial
3.82 10
5.66 10-4
4.27 10
GradMask
1 10
9.8 10
RRR
2.35 10
9.57
RSNA VPC
Classification
9.99 10-3
Masked
2.8 10
Actdiff'
x.1 10
1.6
Adversarial
9.83 10-4
10
10:1
1.16 x 10-4
GradMask
1 10-2
6.57
RRR
9.99 x 10
x 10
RSNA No VPC
Classification
5.5 10-4
Masked
2.84 10 10
Actdiff
1 10-4
3.88
Adversarial
3.4 10-5
1 10
6:1
2.35 x 10-4
GradMask
1 10
1 10
RRR
3.11 x 10-4
2.3 10-1
Table 3: Best hyperparameters found using for each hyperparameter search. Discriminator iteration ratios presented as the update ratios the discriminator encoder.
14

---

Published as a conference paper at ICLR 2021
A.5 VISUALIZATIONS OF INPUT GRADIENTS, INTEGRATED GRADIENTS, AND SAALUD BASED ATTRIBUTION MAPS
OCCLUSION
Image
A. Baseline
B: Masked
Adversarial
D: Actot
Grak
ARR
Synthenc
Image
A: Baseline
B: Masked
C. Advvensaiial
D: ActD
E: GradMask
F: RRR
Synthetie
Baseline
Masieed
Adverssrial
D: ActD
E: Grad
RAR
Synthetic o
Figure A.3: Mean input gradients (G; top), integrated gradients (IG; middle), and occlusion-based (O; bottom) saliency maps from 500 randomly-selected Drest images in the Synthetic dataset after taking the absolute value, smoothing, and thresholding at the 50th percentile across all 10 seeds. The first image shows the mean input image of the model over all examples shown, with the mask overlaid in red.
Image
A: Baseline
B: Masked
Adversarial
Di Actiffff
E: Grad
F: ARR
With SPC: G
Image
A: Baseline
B: Masked
C. Adversarial
Di ActDfff
GradMask
ARA
IG
Adversarial
ActD
GradMaak
Figure A.4: Mean input gradients (G; top), integrated gradients (IG; middle), and occlusion-bart (O: bottom) saliency maps from 500 mandomly-selecter Diest images in the R-Ray dataset with site-pathology correlation (SPC) after taking the absolute value, smoothing, and thresholding at the 50th percentile across all 10 seeds. The first image shows the mean input image of the model all examples shown, with the mask overlaid red.
15

---

Published as a conference paper at ICLR 2021
Baseline
Masixed
Adversarial
D: Acto
Grac
AAR
Image
Image
Figure A.5: Mean input gradients (G: top), integrated gradients (IG; middle), and occlusion- -ased (O; bottom) saliency maps from 500 seddomll-ssleeteee Dtest images the X-Ray dataset without site-pathology correlation (No SPC) after taking the absolute value, smoothing, and thresholding the t0th percentile across all 10 seeds. The first image shows the mean input image the model over all examples shown, with the mask overlaid in red.
Image
Baseline
B: Masiced
Adversaria!
Di ActDurr
GradMaak
ARR
G
Masked
Adversarial
GradMask
With VPC: G
Image
VPC
Figure A.6: Mean input gradients (G; top), integrated gradients (IG; middle), and occlusion-based (O: bottom) saliency maps from 500 randomly-selecte view-pathor view-pathology correlation (VPC) after taking the absolute value, smoothing, and thresholding at thth percentile across 10 seeds. The first image shows the mean input image of the model over all examples shown, with the mask overlaid in red.
Dtest images in the RSNA dataset with
16

---

Published as a conference paper at ICLR 2021
Image
A. Baseline
B: Masked
Adversarial
Di ActiCff
Grak
ARR
No G
Adversarial
GradMask
Image
Baseline
Figure A.7: Mean input gradients (G: top), integrated gradients (IG; middle), and occlusion- based (O: bottom) saliency maps from 500 sandomly-selecter Dtest images in the RSNA dataset with no view-pathology correlation (No VPC) after taking the absolute value, smoothing, and thresholding the 50th percentile across all 10 seeds. The first image shows the mean input image the model over all examples shown, with the mask overlaid in red.
17

---

Published as a conference paper at ICLR 2021
A.6 SALIENCY MAPS FOR INCORRECT AND CORRECT PREDICTIONS
Here, we visualize mean gradients from samples the model gets correct (top) vs. incorrect (bottom) for 100 samples from each seed for total of 1000 samples. While the synthetic dataset shows obvious differences for correct and incorrect samples, no such pattern is obvious for the real world datasets.
Image
Baseline
Masked
C: Adversarial
D: ActDir
GradMask
Synthatic: ls
Image
A: Baseline
B: Masked
C: Adversarial
D: ActDitr
E: GradMask
F. RRR
Synthotic: iG
Figure A.8: Mean integrated gradients attribution maps from 500 randomly-selected Drest images in the synthetic dataset, showing correct predictions (top) separately from incorrect predictions (bot- tom), after taking the absolute value, smoothing, and thresholding at the 50th percentile across all 10 seeds. The first image shows the mean input image of the model over all examples shown, with the mask overlaid in red.
Image
A: Baseline
B: Masked
C: Adversarial
D: ActDiff
GradMask
F: RRR
With SPC. is
Image
A: Baseline
B: Masked
C: Adversarial
D: ActDiff
GradMask
F: RRR
With SPC: lG
Figure A.9: Mean integrated gradients attribution maps from 500 randomly-selected Dtest images in the X-Ray SPC dataset, showing correct predictions (top) separately from incorrect predictions (bottom), after taking the absolute value, smoothing, and thresholding at the 50th percentile across all 10 seeds. The first image shows the mean input image of the model over all examples shown, with the mask overlaid in red.
18

---

Published as a conference paper at ICLR 2021
Image
No SPC: I6
Image
A: Bascline
B: Masked
C: Adversarial
D: ActDiff
GradMask
F: RRR
No SPC. Is
Figure A.10: Mean integrated gradients attribution maps from 500 randomly-selectee Dtest images the X- X-Ray No SPC dataset, showing correct predictions (top) separately from incorrect predictions (bottom), after taking the absolute value, smoothing, and thresholding at the 50th percentile across all seeds. The first image shows the mean input image of the model over examples shown, with the mask overlaid in red.
Bascline
Masked
Adversarial
ActDiff
GradMask
F: RRR
With VPC: ls
Image
GradMask
With VPC: ls
Figure A.II: Mean integrated gradients attribution maps from 500 randomly-selected Drest images in the RSNA VPC dataset, showing correct predictions (top) separately from incorrect predictions (bottom), after taking the absolute value, smoothing, and thresholding at the 50th percentile across all 10 seeds. The first image shows the mean input image of the model over all examples shown, with the mask overlaid in red.
19

---

Published as a conference paper at ICLR 2021
Image
Adversarial
ActDiff
GradMask
No VPC: IG
Image
A: Baseline
B: Masked
C: Adversarial
D: ActDiff
GradMask
F: RRR
No VPC. is
Figure 12: Mean integrated gradients attribution maps from 500 randomly-selecterr Dzest images the RSNA No VPC dataset, showing correct predictions (top) separately from incorrect predic- tions (bottom), after taking the absolute value, smoothing, and thresholding at the 50th percentile across all 10 seeds. The first image shows the mean input image of the model over all examples shown, with the mask overlaid in red.
20

---

Published as a conference paper at ICLR 2021
A.7
GENERALIZATION ABILITY VS ATTRIBUTION QUALITY FOR ALL ATTRIBUTION METHODS
Classification Performance Localization Performance (Normal)
Dataset
Synth
0.5
XRay SPC
Test loU
0.4
RSNA VPC
Approach
Baseline
Domain Invariance
Mask
Saliency
0.0
Test AUC
Classification Performance vs. Localization Performance (Integrated)
0.7
Dataset
Synth
0.5
XRay
Test lou
RSNA VPC
0.4
Approach
Baseline
Domain Invariance
0.2
Mask
0.1
Saliency
0.0
0.2
0.4
0.6
0.8
1.0
Test AUC
Classification Performance vs. Localization Performance (Occlude)
Dataset
0.5
Synth
XRay SPC
Test
0.4
RSNA VPC
0.3
Approach
Baseline
0.2
Domain Invariance
Mask
Saliency
Test AUC
Figure A.13: Test AUC vs. Test loU (good saliency) for all seeds evaluated with covariate shift. Models presented grouped by "Approaches": baseline, input masking, domain-invariance, and saliency-based approaches. We present the relationship between classification performance and calization performance using input gradients (top left), integrated gradients (top right) and occlusion methods (bottom) for comparison.
A.8 ACTDIFF LAMBDA EVALUATION
In real-world experiments without covariate shift, found the ActDiff method performed worse than baseline. We suspected this was due to the minimum lambda value in the range considered during hyperparameter tuning ([1 10 10-+10 being too large: the ActDiff model should perform equivalently to the baseline this lambda value is small enough. We tested this on both datasets by running new hyperparamater search with expanded range for the actdiff lambda [1 10-16100
21

---

Published as a conference paper at ICLR 2021
which call the wide search, and report the test set values over 10 seeds in Table 4 While recovered the baseline performance for the No SPC dataset, we were unable to do so on the No VPC dataset. We suspect that the ActDiff penalty is quite strong in the RSNA data due the smaller input masks, making even very small lambda values ActDiff a powerful regularizer of the model, and making the hyperparameter search difficult given fixed compute budget. We conclude that the difficulty of tuning the hyperparamaters in situations where no covariate shift exists in the data is potential downside of this approach.
Dataset
Experiment
Search
Learning Rate
Lambda
AUC
X-Ray
SPC
Baseline
0.70 + 0.05
Normal
5.38 x 10
1 10"1
0.73 +: 0.03
Wide
5.64 x 10-1
0.71 = 0.04
No SPC
Baseline
0.93 + 0.01
Normal
8.92 x 10-3
10-1
0.63 = 0.20
Wide
10
16
0.93 1:
RSNA
VPC
Baseline
0.20 1: 0.03
Normal Wide
x 10
166
0.68 1: 0.05
10-P
9.26 x 10-13
0.71 + 0.04
No VPC
Baseline
0.76 1: 0.02
Normal
1x 10 1x 10-5
3.88
0.60 = 0.02
Wide
1.7 x 10
0.61 + 0.02
Table 4: Hyperparamaters and Test AUC (mean + standard deviation) for the actdiff models with a wide hyperparameter search range on the X- X-Ray SPC and RSNA VPC datasets across 10 seeds.
A.9 INDIVIDUAL SAMPLES
GredMask
BAR
+
++
Figure A. 14: Randomly-selecte images for the synthetic dataset from 5 trained models (1 per row)
for each training method, computed using integrated gradients.
22

---

Published as a conference paper at ICLR 2021
Raseline
Macked
Adversarial
ActDif
GradBtakk
with SPC IG
Figure A.15: Randomly-selece images for the X-Ray SPC dataset from 5 trained models (1 per row) for each training method, computed using integrated gradients.
Bascline
Masted
C: Adversarial
ActDitt
GradMtakk
RRA
No SPC::
Figure A.16: mondomly-selected images for the X-Ray No SPC dataset from trained models (1 per row) for each training method, computed using integrated gradients.
23

---

Published as a conference paper at ICLR 2021
Raseline
Masked
C: Adversarial
ActDif
Graditakk
VIAE VPCC
Gandomly-selecte images for the RSNA VPC dataset from 5 trained models (1 per row) for each training method, computed using integrated gradients.
Figure A.17:
Bascline
Masied
Adversarial
ActDirt
GradMlakk
RRA
Figure A.18: Randomly- selected images for the RSNA No VPC dataset from trained models (1 per row) for each training method, computed using integrated gradients.
24

---

Published as a conference paper at ICLR 2021
A.10 ANALYSIS OF MODEL PERFORMANCE WITH EQUIVALENT SKEW TRAIN AND VALIDATION SETS
The hyperparameter searches in this paper rely on the existence of covariate shift between the train- ing and validation distributions: generalization performance to the validation set is used to select the regularization lambdas which encourage the model to ignore the confounder. In the typical machine learning setup, the training and validation set are drawn IID from the same distribution. As we saw on the baseline experiments, where there butions were the same, the regularization terms can hurt performance and hyperparamater search will seek to reduce their impact (since building representations of any confounders will produce good validation performance). This is why in our original experiments it was important that the covariate shift not be equivalent for the train and validation sets, and what we recommend in practice (this is easy to accomplish because the exact sources of the covariate shift do not need to be known, so the data can simply be sampled from some different environment where many back- ground variables are likely to differ from the training distribution). Here, we present the results of seeds run for all models on both the X-Ray SPC and RSNA VPC datasets, where the train and validation sets both exhibited 90/10 skew (exactly matching the skew of the training sets in the original paper), and the test set exhibits a 10/90 skew. We set all hyperparamaters to be the same as found during the search made for the main paper, except for setting the regularization term's 's lambda to for all methods (where applicable) since this number cannot be searched for because there is no determine which lambda works (also see the results in Table
covariate shift in the data and the train/valid distri-
covariate shift in the validation 4
In all cases we observe high validation performance and poor test performance. Interestingly, the method which performs best in both cases is the Masked approach, which has no lambda to tune. For both datasets, it reduces the validation performance in exchange for improved test performance. The gradmask approach also improves test performance in the X-Ray SPC dataset. In total we take these results to suggest that these methods must used in cases where there exists some known or unknown distribution shift between the training and validation sets to set the correct lambda: they should not both be drawn IID from the same underlying distribution.
X-Ray: SPC
Experiment
Valid AUC
Test AUC
Classification
0.96 + 0.00
0.64 + 0.07
Baseline
Masked
0.93 1: 0.04
0.78 :: 0.03
ActDiff
0.97 +: 0.00
0.69 = 0.03
Domain Invariance
Adversarial
0.96 + 0.01
0.63 = 0.03
GradMask
0.98 1 0.00
0.74 + 0.03
Saliency Based
RRR
0.96 + 0.01
0.59 = 0.06
RSNA: VPC
Experiment
Valid AUC
Test AUC
Classification
0.92 + 0.01
0.37 + 0.02
Bascline
Masked
0.57 1 0.12
0.06
ActDiff
0.83 + 0.02
0.33 0.02
Domain Invariance
Adversarial
0.82 1 0.06
0.34 0.02
GradMask
0.92 + 0.00
0.37 = 0.01
Saliency Based
RRR
0.92 + 0.00
0.36 0.01
Table 5: Mean and standard deviation Valid and Test AUC for all methods tested on the X-Ray SPC and RSNA VPC datasets across 5 seeds.
25


Reviews:
[{"review_id": "jyebVotFNOA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "--- Summary ---\n\nThis paper focuses on the confounder problem that spatially-seperated image regions (e.g. shoulders of xray images) might spuriously correlated with the target (e.g. pneumonia). If given a human-labeled region that is deemed important, we can decrease this spuriousness by regularizing the model toward the important region. They not only compare with several existing saliency-based methods (RRR and GradMask), but also propose 2 new methods (ActDiff, Adversarial) inspired from domain adapataion literature that the representation of the classifier should be similar between original image and the masked image (the image that the non-important region is shuffled). They compare in 1 synthetic dataset and 2 xray datasets. They show that (1) these methods (sometimes) hurt generalization when spuriousness does not exist, and (2) the model's saliency map is only weakly correlated with generalization performance, and thus doubting the validtiy of using saliency maps for diagnosing whether a model is overfit to spurious features.\n\n--- Pros ---\n\n1. Very detailed hyperparameter search and lots of repeated run (10) to get good standard deviation\n2. Visualizations of average test images are very intriguing.\n\n--- Major comments ---\n\n1. It seems that when there is covariate shift, the ActDiff performs the best but hurts the performance when no shift exists, but saliency-based methods do not suffer this. But given that you can tune the lambda of the regularization, why could it happen? Can't you just pick lambda close to 0?\n\n2. Some visualization looks suspiciously abnormal. For example, RRR in Figure 5 has a big blur in the middle. Why is that? Also, in Figure 7, Masked focus on the neck to predict but neck is clearly outside of the bounding box. How should we explain this phenomenon? Besides, in Figure 7 upper rows (VPC), Adversarial seems to have better masks by focusing on the lungs, and ActDiff does not. But their IOU is reversed: Adversarial has lower IOU and ActDiff has higher IOU in Table 2. \n\n3. In all the experiments, the validation set is always assumed correct without any background shift. While in real scenario it might not always be easy to access to a validation dataset without any shift. When we instead only have a biased validation dataset, which method will perform better? Will the result change?\n\n--- Minor Comments ---\n\n1. Some hyperparameters for visualization like Gaussian smoothing and thersholding might need justifications. Especially the maximum value capped in 50th percentile seems to be a bit excess.\n\n2. When visualizing the gradients (Fig. 5, 6, 7), maybe we should only include images that model predicts correctly? The gradient of extremely wrong predictions might not be very meaningful.\n\n3. The Figure 8 shows the scatter plot between Test AUC and IOU. It seems Synthetic and Xray SPC have much higher IOU overall. Maybe because their bounding box or confounder is location-wise fixed while only the RSNA VPC has varying bounding boxes?\n\n4. The masked baseline perform much worse in No VPC with AUC=0.5 which is random guessing. But it does not happen in other datasets. Maybe it's because the VPC has smaller bounding box and thus only access to such region is too difficult?\n\n5. The lambda should be included in all the equations (eq. 1 to 5).\n\n6. The final hyperparameter should be reported in the appendix.\n\n--- Overall evaluations ---\n\nOverall I like this paper. The hyperparameter tuning is very thorough. The conclusion is good that great saliency map does not mean better accuracy and vice versa. The experimental results are a bit unsatisfying that no real data is improved. And sometimes the simple masked baseline outperform others. I am happy to increase my score if my major concerns are addressed.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_B8WeI38qMP", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Updated recommendation after major changes to the submission: thank you for addressing my comments.\n\nShort summary\n----------------------------\nThe authors investigate the relationship between model generalization under distribution shifts and attribution techniques. They hypothesize that imposing “better” attributions in a model (which they define as being more aligned to a mask selected by domain experts) would increase model generalizability. However, they observe that such constraints hurt the performance under no shift, and do not necessarily lead to increased performance or “better” feature attribution maps under shift.\n\nStrengths\n-----------------------------\nThis work investigates 3 datasets: one synthetic dataset where the effects are well-controlled for, as well as 2 manipulations of real-world data in medical imaging. It includes different and recent techniques to constraint model learning by masked representations, and does not make bold claims about the results.\n\nWeaknesses\n----------------------------\nThere are however a few weaknesses that represent major concerns to me:\n1) The main assumption underlying this work is that improving feature attributions will help generalizability. I however do not think that this is the message that was relayed in past publications on the topic (contrarily to what is mentioned in the introduction) and do not think that “better” attributions is sufficient for better generalizability. My reading on this topic is that attribution methods can help in highlighting confounding factors when models do not generalize under distribution shifts. Making this a sufficiency condition is a step that is not well-motivated to me.\n2) The reason why I do not believe this assumption is valid is because attribution methods have been shown to not correctly represent model’s decisions and to be sensitive to different factors (including shift-variance, Kindermans et al., 2017, or being susceptible to adversarial perturbations). In addition, different attribution methods will likely display different patterns.\n3) Which is why I am wondering whether other attributions, which are more theoretically grounded (e.g. integrated gradients [Sundararajan et al., 2017] or more recent gradient-based techniques like DeepLIFT, or from non-gradients based techniques like SHAP or occlusion) were investigated.\n4) The authors mention that masks represent a “good” attribution map. However, there is no guarantee that features in those masks are not affected by distribution shifts. This should be discussed as a limitation of mask-based regularizers.\n5) The authors seem to have missed that the “masked” trained classifier also highlights the confounder on the synthetic data, despite it correctly predicting the outputs. I believe this is related to the choice of saliency maps, as “raw” gradients do not “explain” the decision of the model, i.e. it does not display the effect the features have between a bad or neutral decision and the current prediction. For this reason, I would suggest using integrated gradients or another technique.\n6) The purpose of the study is quite vague and its results and conclusions lack of depth and reflexion.\n\nNovelty:\n------------------------\nThe authors propose two novel methods to constrain models by using masked representations. While in theory they seem interesting (e.g. activations have been successfully used for OOD detection), the results obtained vary per dataset and there is no clear advantage in terms of model generalization to be using their technique over other methods.\nI feel that the main assumption is not well-grounded, and hence, to me, novelty does not reside in the field tackled.\n\nClarity: \n----------------------\nMainly, I found the paper well written and the experiments clear. I would suggest some proof-reading (see minor comments, some repeated or missing words). Mostly I would suggest to revise the introduction as it wasn’t clear to me what the purpose of the study was until well into the experiments. I also found that the introduction is not well matched to the main message of the paper (e.g. it does not mention mask-related constraints).\n\nRigor:\n------------------------\nI found that the comparison between the proposed technique and the literature was well explained and sufficient. Hyper-parameters were described and confidence intervals were provided on all results. I enjoyed the fact that three datasets were included. Overall, I think the experiments were well executed.\n\nDetailed comments:\n-----------------------------\n- “A critical assumption underlying these aforementioned works is that properties of the saliency map are indicative of generalization performance.” I do not agree with that statement given my major concerns (1).\n- Intro: masking is mentioned in the key contributions but not before and no justification is used. Such masking also assumes that the shifts across train and test distributions do not impact features in this mask. This seems like a strong hypothesis to me, especially when considering e.g. different imaging sites, or image resolutions.\n- It is unclear whether the goal is to obtain models that generalize better or whether it is to obtain feature attribution maps that are consistent with expert input. These two goals can be misaligned, as displayed in the results and should not be conflated.\n- actdiff method: can lambda be mentioned in the equation? How would such a regularizer perform in a high dimensional layer (curse of dimensionality)? Why use pre-activation outputs? Were any other versions of this formulation tried?\n- How were the masks defined? Is there variability per sample (e.g. different experts) in their definition?\n- The authors refer to the synthetic dataset as the changes in the chest x-ray but then have a separate synthetic dataset. The language is confusing and datasets should be defined before being referenced to.\n- Synthetic data results: the masked model performs best, ignoring the confounder. However, the saliency map reflects that the attributions are high for the confounder. Therefore, I do not see the same “high correlation” between IoU and AUC that the authors mention. In addition, this, to me, reflects the main limitation of attribution maps: they do not reflect the model’s decision. They rather reflect the local effect of a feature on the label (Lipton, 2016, Ancona et al., 2018). I am wondering why the authors select saliency maps compared to e.g. integrated gradients (Sundararajan et al., 2017) or other gradient-based but more mathematically grounded techniques. Given my own experience of attribution techniques, it is likely that using different attribution methods will lead to different conclusions with respect to the correlation between IoU and AUC.\n- I understand the choice in the setup of confounders for both datasets. However I am unsure how this represents real-world settings. For example, training sets might indeed be site-specific, but it would be surprising to me that the test presents the inverse of the confounder. We could for example expect the absence of that confounder (which could be simulated by removing the confounder in the test set for synthetic data), or lower correlations between a feature and the label.\n- ActDiff substantially decreases model performance in the absence of confounders\n- If the goal is to obtain more generalizable models, other techniques could be envisaged when the relationship between label and confounder is not deterministic, like resampling or reweighting. Were these considered?\n\nMinor:\n------------\n- Intro: PAC undefined\n- intro: the authors conflate the behavior of a model and which inductive biases it relies on, with the obtained saliency maps. This is however a complex and, in my opinion, unanswered question.\n- Figure 1: NIH, PC, PA and AP not defined. SPC and VPC are presented succinctly without intuition and we only understand them much later. Maybe this figure should be moved in the results.\n- contributions: point 1 needs proof-reading\n- “out of distribution feature attribution phenomena”: I searched DeGrave et al for this term but could not find it. If not used elsewhere, I would rephrase as this is a confusing formulation: samples/images can be OOD, and these samples can provide feature attributions, but the feature attributions themselves are not OOD.\n- proof-reading of the text is required. Some explanations are poorly framed and can be rephrased (e.g. related works, paragraph 2).\n- Zeiler & Fergus, 2013 (published at ECCV 2014) refers to the work on occlusion, where inputs are masked by a baseline value and the changes in predicted risk is defined as their attribution. While this technique provides one attribution per feature, it is not based on the gradients of the network. This paper also does not refer to the term “saliency”.\n- Formulas 4 and 5, consistency in the formulations is desirable: if showing for binary classification in most cases, it is better to keep this set up for other formulations (even if they could be extended to multiclass), especially as the experiments are run on binary classification. The limitation of GradMask to binary cases could however be mentioned.\n- Figure 5 can be difficult to investigate without colormaps. For instance, it looks to me like the “masked” training model does highlight the confounding factor.", "extra_scores": {}, "extra_text": {}}, {"review_id": "UOe_7ax64i-", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the potential correlation between saliency map values and model generalization ability in image analysis with a focus on medical image. \nThe paper falls well within the scope of the conference.\nIt is overall well written, but it is so crammed with information that, in order to comply with paper length limits, its 'digestion' and \ninterpretation becomes difficult at times (this is clear in the case of images, whose meaning often has to be half-guessed due to lack of details)\nThe problem this study deals with is definitely important in the context of medical decision making on the basis of image, and the problems it pinpoints are more than real (small sample sizes, inter-site heterogeneity, bias due to human intervention in the masking process, etc)\nThe proposal seems sound to me and the experiments convincing. My main qualm concerns their specificity, given that they address a very specific domain.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6L1us4pmrVF", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The reviewed paper explores the relationship between the quality and spatial distribution of the saliency maps produced at inference time and the model's generalization performance. The authors employed a number of existing methods as well as proposed and implemented their own technique (ActDiff) to align saliency maps with causally plausible regions. All methods were applied on synthetic and real-world data in a series of clever experiments, showing little correlation between saliency map spatial alignment and performance on unseen data.\n\nOverall, the paper seems to be technically sound, claims justified, and supported by the evidence presented in tables and figures.  Importantly, the paper raises a very interesting point, challenging the status quo in the field. The manuscript is relatively easy to read and understand. For all these reasons, I vote for accepting. \n\nMajor questions-concerns:\n* Figures 6 and 7 show mean saliency maps from randomly selected test images. Columns of these images show outputs from different algorithms used in this work. My question is why do saliency maps produced by methods such as Masked on test images show significant activity in the regions that were explicitly made useless for training (by randomly shuffling pixels outside masks)? I believe this requires a more elaborate and explicit answer.\n* Perhaps it makes sense to openly recognize that before completely dismissing the validity of using saliency maps for diagnosing overfitting a lot more datasets must be studied as the two presented in the paper, may not be enough to provide conclusive evidence.   \n\nSeveral minor comments and questions:\n* Some acronyms, e.g. PAC on the first page is not defined before being used.\n* Caption for Figure 1 can be made more clear, as there is no explanation of what \"pathology correlation with site/view\" means. It is only later in the text, the authors add that they have intentionally biased positive cases by sampling them mostly from either one site or one view. But before reading this part, the caption remains confusing for the reader.\n* It might be a good idea to reformulate contributions of the paper into nouns instead of verbs e.g. instead of \"Create a dataset\" - \"A dataset\".\n* A sentence from the related work section, namely: \"Zhuang et al. (2019) was additionally designed...\" should be reformulated. \n\nUpdates: Thanks for the authors' response. I believe this paper is valuable for the field and community and therefore I recommend this paper to be accepted.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all agreed on accepting this paper, stating that it makes a compelling point about the usefulness of saliency methods to diagnose generalization.  The reviewers found that the experiments were a strong point and applauded the thorough hyperparameter tuning and re-runs for statistical significance.  One reviewer commented that the paper was too dense with information, so much so as to make it difficult to digest.  However, overall this seems like an interesting paper that is relevant to the community and will hopefully foster some good discussion about the shortcomings and future directions of saliency methods.
Decision: Accept (Poster)