Saliency is a Possible Red Herring When Diagnosing Poor Generalization

1 INTRODUCTION . A fundamental challenge when applying deep learning models stems from poor generalization due to covariate shift ( Moreno-Torres et al. , 2012 ) when the probably approximately correct ( PAC ) learning i.i.d . assumption is invalid ( Valiant , 1984 ) i.e . the training distribution is different from the test distribution . One explanation for this is shortcut learning or incorrect feature attribution , where the model during training overfits to a set of training-data specific decision rules to explain the training data instead of modelling the more general causative factors that generated the data ( Goodfellow et al. , 2016 ; Reed & Marks , 1999 ; Geirhos et al. , 2020 ; Hermann & Lampinen , 2020 ; Parascandolo et al. , 2020 ; Arjovsky et al. , 2019 ; Zhang et al. , 2016 ) . In medical imaging , poor generalization due to test-set distribution shifts are common and this problem is exacerbated by small cohorts . Previous work has hypothesized that this poor generalization is in part due to the presence of confounding variables in the training data such as acquisition site or other image acquisition parameters because attribution maps ( aka saliency maps ; Simonyan et al . ( 2014 ) ) produced by the trained model do not highlight features that a human expert would use to make a diagnosis ( Zech et al. , 2018 ; DeGrave et al. , 2020 ; Badgeley et al. , 2019 ; Zhao et al. , 2019 ; Young et al. , 2019 ) . Previous researchers have made the assumption that saliency maps can demonstrate that the model is not overfit or behaving unexpectedly ( Pasa et al. , 2019 ; Tschandl et al. , 2020 ) . We started this work under the same assumption only to find the contradictory evidence we present in this paper . In this work , we set out to test the hypothesis that models with good generalization properties have attribution maps which only utilize the class-discriminative features to make predictions , by explicitly regularizing the models to ignore confounders using attribution priors ( Erion et al. , 2019 ; Ross et al. , 2017 ) , i.e. , to make predictions using the correct anatomy ( as a doctor would ) . We evaluated whether this regularization would A ) improve out of distribution generalization , and B ) change feature attribution to be more like the attribution priors . If there exists a relationship between the attribution map and generalization performance , we would expect these regularizations to positively impact both generalization and attribution quality simultaneously . To evaluate attribution quality , we define good attribution to be an attribution map that agrees strongly with expert knowledge in the form of a binary mask on the input data . We show that the existing and proposed feature-attribution-aware methods help facilitate generalization in the presence of a train-test distribution shift . However , while feature-attribution-aware methods change the attribution maps relative to baseline , there is no strong correlation between generalization performance and good attribution . This in turn challenges the assumption made in previous works that the “ incorrect ” attribution maps were indicative of poor generalization performance . This suggests that A ) efforts to validate model correctness using attribution maps may not be reliable , and B ) that efforts to control feature attribution using masks on the input may not function as expected . All code and datasets for this paper are publicly available1 . Our contributions include : • A synthetic dataset that encourages models to overfit to an easy to represent confounder instead of a more complicated counting task . • Two new tasks constructed from open medical datasets which have a correlation between the pathology and either imaging site ( site pathology correlation ; SPC ) or view ( view pathology correlation ; VPC ) , and we manipulate the nature of this correlation differently in the training and test distributions to create a distribution shift ( Figure 5 ) , introducing confounding variables as observed in previous work ( Zhao et al. , 2019 ; DeGrave et al. , 2020 ) . • Evaluation of existing methods for controlling feature attribution using mask information ; right for the right reasons ( RRR ; Ross et al . ( 2017 ) ) , GradMask ( Simpson et al. , 2019 ) , and adversarial domain invariance ( Tzeng et al. , 2017 ; Ganin & Lempitsky , 2015 ) . • A new method for controlling feature attribution based on minimizing activation differences between masked and unmasked images ( ActDiff ) . • Evaluate the relationship between generalization improvement and feature attribution in real- life out of distribution generalization tasks with traditional classifiers . 2 RELATED WORK . It is a well-documented phenomenon that convolutional neural networks ( CNNs ) , instead of building object-level representations of the input data , tend to find convenient surface-level statistics in the training data that are predictive of class ( Jo & Bengio , 2017 ) . Previous work has attempted to reduce the model ’ s proclivity to use confounding features by randomly masking out regions of the input ( DeVries & Taylor , 2017 ) , forcing the network to learn representations that aren ’ t dependent on a single input feature . However , this regularization approach gives no control over the kinds of representations learned by the model , so we do not include it in our study . Recently , multiple approaches have proposed to control feature representations by penalizing the model for producing saliency gradients outside of a regions of interest indicating the classdiscriminative feature ( Simpson et al. , 2019 ; Zhuang et al. , 2019 ; Rieger et al. , 2019 ) . These approaches were introduced by Right for the Right Reasons ( RRR ) , which showed impressive improvements in attribution correctness on synthetic data ( Ross et al. , 2017 ) . The follow-up work has generally demonstrated a small improvement in generalization accuracy on real data , and much more impressive results on synthetic data . Another feature attribution-aware regularization approach additionally dealt with class imbalances by increasing the impact of the gradients inside the region of interest of the under-represented class Zhuang et al . ( 2019 ) . One alternative to saliency-based methods , which can be noisy due to the ReLU activations allowing irrelevant features to pass through the activation function ( Kim et al. , 2019 ) , would be to leverage methods that aim to produce domain invariant features in the latent space of the network . These methods regularize the network such that the latent representations of two or more “ domains ” are encouraged to be as similar as possible , often by minimizing a distance metric or by employing an adversary that is trained to distinguish between the different domains ( Kouw & Loog , 2019 ; Ganin & Lempitsky , 2015 ; Tzeng et al. , 2015 ; Liu & Tuzel , 2016 ) . In this work , we view the masked version of the input as the training domain and the unmasked version of the input as the test domain , and compare these approaches with saliency-based approaches for the task of reducing the model ’ s reliance on confounding features . To the best of our knowledge , these strategies have not been tried to control feature attribution . 1https : //github.com/josephdviviano/saliency-red-herring 3 METHODS . Domain Invariance with the Activation Difference and Adversarial Loss : Leveraging ideas from domain adaptation , we introduce two methods designed to make the model invariant to features arising from outside of the attribution priors . The first is the embarrassingly simple Activation Difference ( ActDiff ) approach , which simply penalizes , for each input , the L2-normed distance between the masked and unmasked input ’ s latent representations . This loss is most similar to a style transfer approach which transforms a random noise image into one containing the same layerwise activations as some target image ( Gatys et al. , 2015 ) to apply a visual aesthetic to some input semantic contents , and encourages the network to build features which appear inside the masked regions even though it always sees the full image during training . Therefore we minimize Lact = ∑ ( Xmasked , x ) ∈D Lclf + λact||ol ( xmasked ) − ol ( x ) ||2 , ( 1 ) where ol ( x ) are the pre-activation outputs for layer l of the n-layer encoder f ( x ) when the network is presented with the original data x , ol ( xmasked ) are the pre-activations outputs for layer l when presented with masked data xmasked , and Lclf is the standard cross entropy loss . We define xmasked = x·xseg+shuffle ( x ) · ( 1 − xseg ) where background pixels are shuffled uniquely for each presentation to the network in order to destroy any spatial information available in those regions of the image without introducing any consistent artefacts into the image or shifts in the distribution of input intensities . This method is at a high level similar to using maximum mean discrepancy ( MMD ) for domain adaptation ( Baktashmotlagh et al. , 2016 ; 2013 ) where instead of minimizing the distance between the means of the two domain distributions , we instead minimize the distance between the features directly . One could replace the L2 norm with any Lk norm , and choices of k < 2 might be useful when regularizing larger latent representations as L2 distances collapse to a constant value in extremely high dimensional spaces due to the curse of dimensionality ( Aggarwal et al. , 2001 ) . We found during experimentation that regularizing the pre-activations led to better results at the cost of longer time to convergence , perhaps because the L2 norm is more effective when the feature vectors are not sparse , but we leave this conjecture to future work . The second approach we explore employs a discriminator D ( · ) optimized to distinguish between latent representations arising from passing xmasked or x through the encoder f ( · ) ( Tzeng et al. , 2017 ; Goodfellow et al. , 2014 ) . Simultaneously , we optimize the encoder f ( · ) to fool the discriminator and still produce representations that are predictive of the output class . Therefore , D and f are optimized using the LD and Lf respectively : LD = λdisc ( Exmasked [ logD ( f ( xmasked ) ) ] + Exmasked [ log ( 1−D ( f ( x ) ) ) ] ) ( 2 ) Lf = Lclf + λdisc ( Ex [ log ( 1−D ( f ( xmasked ) ) ) ] + Ex [ logD ( f ( x ) ) ] ) ( 3 ) This approach is similar to the one employed in ( Janizek et al. , 2020 ) , where the authors encouraged the model to be invariant to the view of the X-Ray ( Lateral vs PA ) , and here ‘ view ’ denotes whether the image is masked or not . D ( · ) had three fully-connected hidden layers of size 1024 before outputting a binary prediction . To facilitate the stability of training , we updated D ( · ) more frequently than the encoder treating this ratio as a hyperparameter . We optimized D ( · ) independently using Adam with a distinct learning rate , and applied spectral normalization to the hidden layers of D ( · ) . Direct Attribution Control with the Right for the Right Reasons and GradMask Loss : These input gradient attribution regularizers directly control which regions of the input are desirable for determining the class label by penalizing saliency outside of a defined input mask . The most basic gradient based “ input feature attribution ” can be calculated as ∂|ŷi|∂x for each input x ( Simonyan et al. , 2014 ; Lo et al. , 2015 ) . In the binary classification case , RRR ( Ross et al. , 2017 ) calculates saliency of the summed log probabilities of the 2 output classes with respect to the input x , Lrrr = ∑ ( xseg , x ) ∈D Lclf + λrrr · ∂ ( log ( p̂0 ) + log ( p̂1 ) ) 2 ∂x · ( 1− xseg ) , ( 4 ) where ( 1 − xseg ) is a binary mask that covers everything outside the defined regions of interest . The numerator in the RRR loss can be extended to the multi-class case by substituting ∑K k log ( p̂k ) . GradMask ( Simpson et al. , 2019 ) similarly calculates saliency using the contrast between the two output logits , and is only defined for the binary classification case , Lgradmask = ∑ ( xseg , x ) ∈D Lclf + λgradmask · ∣∣∣∣∂ |ŷ0 − ŷ1|∂x · ( 1− xseg ) ∣∣∣∣ 2 , ( 5 ) where ŷ0 and ŷ1 are the predicted outputs for our two classes . Classify Masked : We evaluated the effect imposing the attribution prior by simply training a model using masked data ( and evaluating it using unmasked data ) as a control experiment . The data was masked by shuffling the pixels outside of the mask during training , as was done for the domain invariance experiments to produce xmasked . We refer to these experiments as Masked .

Summary: The reviewed paper explores the relationship between the quality and spatial distribution of the saliency maps produced at inference time and the model's generalization performance. The authors employed a number of existing methods as well as proposed and implemented their own technique (ActDiff) to align saliency maps with causally plausible regions. All methods were applied on synthetic and real-world data in a series of clever experiments, showing little correlation between saliency map spatial alignment and performance on unseen data.