Beyond Faithfulness: A Framework to Characterize and Compare Saliency Methods

1 INTRODUCTION . As machine learning ( ML ) systems are increasingly deployed into real-world contexts , stakeholder interviews ( Tonekaboni et al. , 2019 ; Bhatt et al. , 2020 ) , design best practices ( Amershi et al. , 2019 ) , and legal frameworks ( European Commission , 2018 ) have underscored the need for explainability . Saliency methods — a class of attribution methods that aim to identify features in an input that were important to a trained ML prediction — are frequently used to provide explanations . However , each method operates differently and , thus , multiple methods can produce seemingly varying explanations for the same model and input . How , then , should we reason about choosing and comparing methods for a particular application ? Prior work has suggested that “ faithfulness , ” or how accurately a saliency result reflects the underlying model , is a desideratum for these methods ( Li et al. , 2021 ; Ding & Koehn , 2021 ; Tomsett et al. , 2020 ; Adebayo et al. , 2018 ) . In this paper , however , we argue that faithfulness is not a productive goal for saliency methods — by design , they can not offer a complete and accurate reflection of a model ’ s behavior , akin to a printout of model weights . Rather , we frame saliency methods as abstractions of model behavior that selectively preserve and necessarily sacrifice information in service of human-centric goals such as simplicity and understandability . With this framing , we propose a nine-dimensional framework to characterize and compare saliency methods . These dimensions fall into three categories , corresponding to different parts of the interpretation process : methodology , or how the saliency is computed ; sensitivity , or relationships between the saliency and the underlying model or input ; and perceptibility , or how an end-user perceives the saliency . These dimensions decompose a singular notion of faithfulness into more granular units that can be reasoned about individually : by situating methods along these dimensions , we can surface their relative strengths , limitations , and differences to better understand trade offs that are latent in their design . In doing so , we demonstrate how our framework allows us to develop “ saliency 0 cards ” ( akin to “ model cards ” ( Mitchell et al. , 2019 ) and “ datasheets ” ( Gebru et al. , 2018 ) ) to document individual methods and to better contextualize saliency results . Moreover , using a concrete example of ML-based radiology diagnostic systems , we show how downstream stakeholders can use our framework to weigh tradeoffs and choose a task-appropriate method . Finally , we show how our framework identifies compelling opportunities for future work including exploring understudied dimensions and developing new metrics that target particular dimensions . 2 RELATED WORK : SALIENCY METHODS AND THEIR EVALUATIONS . Saliency methods ( sometimes referred to as feature attribution methods ) produce explanations for an ML model ’ s output . Given an input , saliency methods compute an importance score for each input feature describing its influence on the model ’ s output . Existing categorizations for saliency methods have focused on algorithmic properties ( gradient or perturbation-based , path-attribution or gradientonly ) Molnar ( 2019 ) . Our framework aims to categorize a broader range of important characteristics . We apply our framework to a variety of common saliency methods listed in Table 1 . Evaluations of saliency methods have primarily focused on how accurately their results represent model behavior , often referred to as faithfulness . A growing body of work has identified failures of some methods such as susceptibility to adversarial perturbations ( Ghorbani et al. , 2019 ) , lack of neuron discriminativity ( Mahendran & Vedaldi , 2016 ) , and a predisposition towards input recovery ( Nie et al. , 2018 ; Adebayo et al. , 2018 ) . Other work has proposed proxy tests that measure different aspects of faithfulness . Adebayo et al . ( 2018 ) recommend model randomization and data label randomization tests , and Kindermans et al . ( 2019 ) test whether constant input shifts affect saliency results . Samek et al . ( 2016 ) judge saliency methods by iteratively replacing features that have high importance values with random noise and measuring how much the output changes . While these tests quantitatively analyze saliency methods , Tomsett et al . ( 2020 ) found they can produce inconsistent rankings . An alternate line of work has looked to break faithfulness down into measurable axioms . Sundararajan et al . ( 2017 ) propose implementation invariance ( a saliency method should produce the same output on functionally equivalent models ) , sensitivity ( a saliency method should give importance to a feature if and only if changing it leads to a different output ) , and linearity ( if a model is the composition of two sub-models , the saliency method ’ s output for the model should be the weighted sum of its outputs for each sub-model ) . Sundararajan et al . ( 2017 ) and Shrikumar et al . ( 2017 ) both posit a saliency method should exhibit completeness — a saliency method ’ s attributions should sum to the difference between the model ’ s output on the input and the model ’ s output on a neutral input . Shrikumar et al . ( 2017 ) and Montavon et al . ( 2018 ) claim saliency methods should output a continuous function . Montavon et al . ( 2018 ) also propose the axiom of selectivity — a saliency method should distribute importance to features that have the greatest impact on the model ’ s output . Finally , Kindermans et al . ( 2019 ) state that a saliency method should be invariant to constant transforms . While axioms are presented as constraints that all saliency methods should attain , the axes in our framework describe attributes of saliency methods that can be traded off to select the best method for different use cases . To situate saliency methods along each axis , we utilize existing tests ( Kindermans et al. , 2019 ; Adebayo et al. , 2018 ) ; however , our framework surfaces the need for additional evaluations , since existing tests do not fully describe each axis and have not assessed all methods . 3 NINE DIMENSIONS TO CHARACTERIZE & COMPARE SALIENCY METHODS . To distill a language to characterize and compare saliency methods , we treat saliency methods as abstractions of the underlying model behavior . That is , to explain model behavior in a humanunderstandable format ( e.g. , a feature attribution heatmap ) , we must necessarily abstract away some detail and concreteness . A saliency method , by design , is not as precise a reflection of model behavior as a printout of the weights , but it is much more human-interpretable . The idea of abstraction recurs in other parts of the machine learning pipeline . For example , “ test accuracy ” is an abstraction of model performance : it does not capture all aspects of model performance but is nevertheless a convenient approximation . Abstractions selectively preserve information and can be combined to arrive at a more complete picture of the underlying phenomenon — i.e. , combining test accuracy with AUROC to more fully understand model performance . This framing reveals that a single saliency method can not fully explain model behavior . Instead , it is necessary to understand different methods ’ strengths and limitations to choose an appropriate method for the model , domain , and task . Our framework defines nine dimensions that describe what information saliency methods abstract , and how this abstraction is computed . We group these dimensions into three categories that map to different stages of the interpretability process ( see Figure 1 ) . Methodology covers the model , input , and saliency method , and dimensions under this category describe how the saliency method operates . Sensitivity goes one stage further , with dimensions that describe the relationship between the saliency output and the model or input . Perceptibility covers the final stages of the process , with dimensions that describe how users perceive the saliency output and any associated visualizations . These dimensions allow us to decompose faithfulness into aspects that can be reasoned about individually . For instance , Figure 2 demonstrates how our dimensions allow us to situate saliency methods in relation to one another to understand their strengths , limitations , and differences . In the following subsections , we describe and provide an illustrative example of each dimension . 3.1 METHODOLOGY : HOW SALIENCY METHODS OPERATE . Determinism Some saliency techniques are non-deterministic , and running them with different random seeds can produce significantly different outputs . For instance , methods such as RISE or SHAP rely on randomly-generated masks or coalitions . The output from a non-deterministic method represents only one example from a potentially high-variance distribution , and may be afforded authority that does not account for its inherent uncertainty . Although we may never explicitly choose a method for its non-determinism , we may nevertheless accept non-determinism in service of other dimensions such as increased model agnosticism ( e.g. , RISE ) , perceptual correspondence ( e.g. , SHAP ) , or minimality ( e.g. , MP ) . Example . Consider a model assisting dermatologists diagnosing skin cancer . In Figure 3 ( a ) , we show that two runs of a non-deterministic method ( LIME ) on a melanoma prediction model result in different saliency maps . These variations may have significant consequences given small areas of the lesion or surrounding skin may be integral to the diagnosis . Visualizing and interpreting multiple runs together is time-consuming and confusing , but only looking at one run could skew a clinician ’ s judgment . Thus , in this case , it might make sense to prioritize a method with deterministic outputs . Hyperparameter Dependence Hyperparameter dependence captures how many hyperparameters or design decisions the user must specify to run a particular method , and how sensitive it is to them . If there are not sufficient resources or expertise to devote to hyperparameter optimization , simply using default values can lead to misleading results . Similarly , misleading or confusing results might arise if the hyperparameters were chosen based on a particular dataset but deployed in a setting in which there is a significant distribution shift . In situations like this , it makes sense to prioritize methods with low hyperparameter dependence . In other cases , where developers have dedicated appropriate time and resources to tuning , it might be preferable to use a method dependent on hyperparameters ( e.g. , IG , SmoothGrad ) because it improves other dimensions ( e.g. , minimality ) . Example . IG computes feature importances by interpolating between a “ meaningless ” baseline input and the actual input , accumulating the gradients at each step . A common practice is to use a baseline value of all zeroes ; however , in some cases , a zero baseline can be misleading and potentially harmful . Take a model trained to predict bone fractures from x-ray images . Fractures often appear as a black line in the bone , but , since they have the same pixel value as the baseline ( zero ) , the IG result will indicate that they are not important . In Figure 3 ( b ) , we show a similar example where the choice of baseline ( black , white ) has a strong effect on the saliency map . Choosing an appropriate baseline requires time and a deep understanding of the data and method ; if this is not available in a particular application , it is better to prioritize a saliency method without important hyperparameters . Model Agnosticism Model agnostic methods ( i.e. , SHAP ) treat the underlying model as a blackbox , relying only on its input and output . On the other hand , model-dependent methods need access to various model internals , and may have specific requirements , such as differentiability ( i.e. ,

Summary: This paper proposes a framework to evaluate saliency-based explanations for neural network decisions. The framework evaluates each method along a number of different dimensions (for example, reporting whether the saliency map output is fixed or stochastic, or the degree to which the output depends on hyperparameters). The proposed framework is not actually implemented or tested in this paper.