Beyond Faithfulness: A Framework to Characterize and Compare Saliency Methods

1 INTRODUCTION . As machine learning ( ML ) systems are increasingly deployed into real-world contexts , stakeholder interviews ( Tonekaboni et al. , 2019 ; Bhatt et al. , 2020 ) , design best practices ( Amershi et al. , 2019 ) , and legal frameworks ( European Commission , 2018 ) have underscored the need for explainability . Saliency methods — a class of attribution methods that aim to identify features in an input that were important to a trained ML prediction — are frequently used to provide explanations . However , each method operates differently and , thus , multiple methods can produce seemingly varying explanations for the same model and input . How , then , should we reason about choosing and comparing methods for a particular application ? Prior work has suggested that “ faithfulness , ” or how accurately a saliency result reflects the underlying model , is a desideratum for these methods ( Li et al. , 2021 ; Ding & Koehn , 2021 ; Tomsett et al. , 2020 ; Adebayo et al. , 2018 ) . In this paper , however , we argue that faithfulness is not a productive goal for saliency methods — by design , they can not offer a complete and accurate reflection of a model ’ s behavior , akin to a printout of model weights . Rather , we frame saliency methods as abstractions of model behavior that selectively preserve and necessarily sacrifice information in service of human-centric goals such as simplicity and understandability . With this framing , we propose a nine-dimensional framework to characterize and compare saliency methods . These dimensions fall into three categories , corresponding to different parts of the interpretation process : methodology , or how the saliency is computed ; sensitivity , or relationships between the saliency and the underlying model or input ; and perceptibility , or how an end-user perceives the saliency . These dimensions decompose a singular notion of faithfulness into more granular units that can be reasoned about individually : by situating methods along these dimensions , we can surface their relative strengths , limitations , and differences to better understand trade offs that are latent in their design . In doing so , we demonstrate how our framework allows us to develop “ saliency 0 cards ” ( akin to “ model cards ” ( Mitchell et al. , 2019 ) and “ datasheets ” ( Gebru et al. , 2018 ) ) to document individual methods and to better contextualize saliency results . Moreover , using a concrete example of ML-based radiology diagnostic systems , we show how downstream stakeholders can use our framework to weigh tradeoffs and choose a task-appropriate method . Finally , we show how our framework identifies compelling opportunities for future work including exploring understudied dimensions and developing new metrics that target particular dimensions . 2 RELATED WORK : SALIENCY METHODS AND THEIR EVALUATIONS . Saliency methods ( sometimes referred to as feature attribution methods ) produce explanations for an ML model ’ s output . Given an input , saliency methods compute an importance score for each input feature describing its influence on the model ’ s output . Existing categorizations for saliency methods have focused on algorithmic properties ( gradient or perturbation-based , path-attribution or gradientonly ) Molnar ( 2019 ) . Our framework aims to categorize a broader range of important characteristics . We apply our framework to a variety of common saliency methods listed in Table 1 . Evaluations of saliency methods have primarily focused on how accurately their results represent model behavior , often referred to as faithfulness . A growing body of work has identified failures of some methods such as susceptibility to adversarial perturbations ( Ghorbani et al. , 2019 ) , lack of neuron discriminativity ( Mahendran & Vedaldi , 2016 ) , and a predisposition towards input recovery ( Nie et al. , 2018 ; Adebayo et al. , 2018 ) . Other work has proposed proxy tests that measure different aspects of faithfulness . Adebayo et al . ( 2018 ) recommend model randomization and data label randomization tests , and Kindermans et al . ( 2019 ) test whether constant input shifts affect saliency results . Samek et al . ( 2016 ) judge saliency methods by iteratively replacing features that have high importance values with random noise and measuring how much the output changes . While these tests quantitatively analyze saliency methods , Tomsett et al . ( 2020 ) found they can produce inconsistent rankings . An alternate line of work has looked to break faithfulness down into measurable axioms . Sundararajan et al . ( 2017 ) propose implementation invariance ( a saliency method should produce the same output on functionally equivalent models ) , sensitivity ( a saliency method should give importance to a feature if and only if changing it leads to a different output ) , and linearity ( if a model is the composition of two sub-models , the saliency method ’ s output for the model should be the weighted sum of its outputs for each sub-model ) . Sundararajan et al . ( 2017 ) and Shrikumar et al . ( 2017 ) both posit a saliency method should exhibit completeness — a saliency method ’ s attributions should sum to the difference between the model ’ s output on the input and the model ’ s output on a neutral input . Shrikumar et al . ( 2017 ) and Montavon et al . ( 2018 ) claim saliency methods should output a continuous function . Montavon et al . ( 2018 ) also propose the axiom of selectivity — a saliency method should distribute importance to features that have the greatest impact on the model ’ s output . Finally , Kindermans et al . ( 2019 ) state that a saliency method should be invariant to constant transforms . While axioms are presented as constraints that all saliency methods should attain , the axes in our framework describe attributes of saliency methods that can be traded off to select the best method for different use cases . To situate saliency methods along each axis , we utilize existing tests ( Kindermans et al. , 2019 ; Adebayo et al. , 2018 ) ; however , our framework surfaces the need for additional evaluations , since existing tests do not fully describe each axis and have not assessed all methods . 3 NINE DIMENSIONS TO CHARACTERIZE & COMPARE SALIENCY METHODS . To distill a language to characterize and compare saliency methods , we treat saliency methods as abstractions of the underlying model behavior . That is , to explain model behavior in a humanunderstandable format ( e.g. , a feature attribution heatmap ) , we must necessarily abstract away some detail and concreteness . A saliency method , by design , is not as precise a reflection of model behavior as a printout of the weights , but it is much more human-interpretable . The idea of abstraction recurs in other parts of the machine learning pipeline . For example , “ test accuracy ” is an abstraction of model performance : it does not capture all aspects of model performance but is nevertheless a convenient approximation . Abstractions selectively preserve information and can be combined to arrive at a more complete picture of the underlying phenomenon — i.e. , combining test accuracy with AUROC to more fully understand model performance . This framing reveals that a single saliency method can not fully explain model behavior . Instead , it is necessary to understand different methods ’ strengths and limitations to choose an appropriate method for the model , domain , and task . Our framework defines nine dimensions that describe what information saliency methods abstract , and how this abstraction is computed . We group these dimensions into three categories that map to different stages of the interpretability process ( see Figure 1 ) . Methodology covers the model , input , and saliency method , and dimensions under this category describe how the saliency method operates . Sensitivity goes one stage further , with dimensions that describe the relationship between the saliency output and the model or input . Perceptibility covers the final stages of the process , with dimensions that describe how users perceive the saliency output and any associated visualizations . These dimensions allow us to decompose faithfulness into aspects that can be reasoned about individually . For instance , Figure 2 demonstrates how our dimensions allow us to situate saliency methods in relation to one another to understand their strengths , limitations , and differences . In the following subsections , we describe and provide an illustrative example of each dimension . 3.1 METHODOLOGY : HOW SALIENCY METHODS OPERATE . Determinism Some saliency techniques are non-deterministic , and running them with different random seeds can produce significantly different outputs . For instance , methods such as RISE or SHAP rely on randomly-generated masks or coalitions . The output from a non-deterministic method represents only one example from a potentially high-variance distribution , and may be afforded authority that does not account for its inherent uncertainty . Although we may never explicitly choose a method for its non-determinism , we may nevertheless accept non-determinism in service of other dimensions such as increased model agnosticism ( e.g. , RISE ) , perceptual correspondence ( e.g. , SHAP ) , or minimality ( e.g. , MP ) . Example . Consider a model assisting dermatologists diagnosing skin cancer . In Figure 3 ( a ) , we show that two runs of a non-deterministic method ( LIME ) on a melanoma prediction model result in different saliency maps . These variations may have significant consequences given small areas of the lesion or surrounding skin may be integral to the diagnosis . Visualizing and interpreting multiple runs together is time-consuming and confusing , but only looking at one run could skew a clinician ’ s judgment . Thus , in this case , it might make sense to prioritize a method with deterministic outputs . Hyperparameter Dependence Hyperparameter dependence captures how many hyperparameters or design decisions the user must specify to run a particular method , and how sensitive it is to them . If there are not sufficient resources or expertise to devote to hyperparameter optimization , simply using default values can lead to misleading results . Similarly , misleading or confusing results might arise if the hyperparameters were chosen based on a particular dataset but deployed in a setting in which there is a significant distribution shift . In situations like this , it makes sense to prioritize methods with low hyperparameter dependence . In other cases , where developers have dedicated appropriate time and resources to tuning , it might be preferable to use a method dependent on hyperparameters ( e.g. , IG , SmoothGrad ) because it improves other dimensions ( e.g. , minimality ) . Example . IG computes feature importances by interpolating between a “ meaningless ” baseline input and the actual input , accumulating the gradients at each step . A common practice is to use a baseline value of all zeroes ; however , in some cases , a zero baseline can be misleading and potentially harmful . Take a model trained to predict bone fractures from x-ray images . Fractures often appear as a black line in the bone , but , since they have the same pixel value as the baseline ( zero ) , the IG result will indicate that they are not important . In Figure 3 ( b ) , we show a similar example where the choice of baseline ( black , white ) has a strong effect on the saliency map . Choosing an appropriate baseline requires time and a deep understanding of the data and method ; if this is not available in a particular application , it is better to prioritize a saliency method without important hyperparameters . Model Agnosticism Model agnostic methods ( i.e. , SHAP ) treat the underlying model as a blackbox , relying only on its input and output . On the other hand , model-dependent methods need access to various model internals , and may have specific requirements , such as differentiability ( i.e. ,

Summary: This article proposes a review of "saliency" methods in deep neural networks and rates them according to 9 dimensions: Computational efficiency, determinism, hyperparameters, model agnosticism, input sensitivity, minimality, perceptual correspondence and semantic directness. The author's claim is that this new taxonomy provides a better way for choosing between saliency approaches for the end-user.   Note: What is meant by saliency in this paper is methods, such as Grad-CAM to determine which inputs were more prevalent in a network's decision (the word has different meanings in other sub-fields).