Beyond Faithfulness: A Framework to Characterize and Compare Saliency Methods
Authors: Angie Boggust
Harini Suresh
Hendrik Strobelt
John Guttag
Arvind Satyanarayan
Venue: ICLR 2022 Submitted (2022)
Keywords: saliency methods
interpretability
faithfulness
explainability
attribution
feature importance

Abstract: Saliency methods calculate how important each input feature is to a machine learning model’s prediction, and are commonly used to understand model reasoning. “Faithfulness,” or how fully and accurately the saliency output reflects the underlying model, is an oft-cited desideratum for these methods. However, explanation methods must necessarily sacrifice certain information in service of user-oriented goals such as simplicity. To that end, and akin to performance metrics, we frame saliency methods as abstractions: individual tools that provide insight into specific aspects of model behavior and entail tradeoffs. Using this framing, we describe a framework of nine dimensions to characterize and compare the properties of saliency methods. We group these dimensions into three categories that map to different phases of the interpretation process: methodology, or how the saliency is calculated; sensitivity, or relationships between the saliency result and the underlying model or input; and, perceptibility, or how a user interprets the result. As we show, these dimensions give us a granular vocabulary for describing and comparing saliency methods — for instance, allowing us to develop “saliency cards” as a form of documentation, or helping downstream users understand tradeoffs and choose a method for a particular use case. Moreover, by situating existing saliency methods within this framework, we identify opportunities for future work, including filling gaps in the landscape and developing new evaluation metrics.


Under review as a conference paper at ICLR 2022
BEYOND FAITHFULNESS: A FRAMEWORK TO CHAR- ACTERIZE AND COMPARE SALIENCY METHODS
Anonymous authors Paper under double-blind review
ABSTRACT
Saliency methods calculate how important each input feature is to machine learn- ing model's prediction, and are commonly used to understand model reasoning. "Faithfulness," or how fully and accurately the saliency output reflects the under- lying model, is an oft-cited desidcratum for these methods. However, explanation methods must necessarily sacrifice certain information in service of user-oriented goals such as simplicity. To that end, and akin to performance metries, we frame saliency methods as abstractions: individual tools that provide insight into specific aspects of model behavior and entail tradeoffs. Using this framing, we describe framework of nine dimensions to characterize and compare the properties of saliency methods. We group these dimensions into three categories that map to different phases of the interpretation process: methodology, how the saliency is calculated; sensitivity, or relationships between the saliency result and the under- lying model input; and, perceptibility, or how user interprets the result. As we show, these dimensions give us a granular vocabulary for describing and compar- ing saliency methods for instance, allowing us develop "saliency cards" a form of documentation, or helping downstream users understand tradeoffs and choose method for a particular use case. Moreover, by situating existing saliency methods within this framework, we identify opportunities for future work, includ- ing filling gaps in the landscape and developing new evaluation metrics.
INTRODUCTION
As machine learning (ML) systems are increasingly deployed into real-world contexts, stakeholder interviews [Tonekaooni et al. 2019 Bhatt et al. 2020). design best practices (Amershi et al.. 2019), and legal frameworks (European Commission 2018) have underscored the need for explamnailityy Saliency methods- a class of attribution methods that aim to identify features in an input that were important to a trained ML prediction - are frequently used to provide explanations. However, each method operates differently and, thus, multiple methods produce seemingly varying explana- tions for the same model and input. How, then, should we reason about choosing and comparing methods for particular application?
Prior work has suggested that "faithfulness," or how accurately a saliency result reflects the under- lying model,
desideratum for these methods (Li et al. [02 Ding & Koehn 2021 Tomsett et al. 2020; Adebayo et al. 2018). In this paper, however, we argue that faithfulness is not pro- ductive goal for saliency weethoss-by design, they cannot offer complete and accurate reflection of a model's behavior, akin to printout of model weights. Rather, we frame saliency methods as abstractions of model behavior that selectively preserve and necessarily sacrifice information in service of human-centric goals such as simplicity and understandability.
With this framing, we propose a nine-dimensional framework to characterize and compare saliency methods. These dimensions fall into three categories, corresponding to different parts of the interpre- tation process: methodology, or how the saliency is computed; sensitivity, or relationships between the saliency and the underlying model or input; and perceptibility. how end-user perceives the saliency. These dimensions decompose a singular notion of faithfulness into more granular units sur-
that can be reasoned about individually: by situating methods along these dimensions, face their relative strengths, limitations, and differences to better understand trade offs that are latent in their design. In doing so, demonstrate how our framework allows us to develop "saliency

---

Under review as a conference paper at ICLR 2022
Method
Description
Vanilla Gradients (VG) (Erhan et al.
Computes the gradient of the model's output for the class of interest with respect the input.
2009
Simonyan al. 2013)
Input Gradient
Extends VG by performing element multiplication on the gradient and input features.
Integrated Gradients (IG) (Sundararajan
Interpolates between baseline example and the input, accumulating gradients at each step.
ct al.
2017)
Computes the gradient of the model's output for the class of interest with respect to the last convolutional layer of a CNN.
GradCAM (eelvarauu al.
2017
SmoothGrad (mmilkov et al.
2017)
Averages over saliency results for slightly perturhed/noisy versions of the input.
Guided Backpropagation (Guided BP)
Extends gradient- mudient-based methods by preventing the flow of negative gradients, resulting in attributions from positive paths in network.
(Springenberg et a 22014)
Local Interpretable Model-Agnostie Ex-
Trains simple surrogate model on a localized dataset generated by perturbing the input example, and uses its coefficients importance values.
planations (LIME) (Riheiro et (016)
Shapley Additive Explanations (SHAP)
Computes attribution as a game theory problem where each feature is "player" and the oulput of the model is "payout" distributed amongst them.
(Lundberg & Lee (0177)
Meaningful Perturbations (MP) Fong &
Strategically masks different parts of the input to learn which are important (i.c., lead to the largest change in the output).
Vedaldi 2017)
Sufficient Input Subsets (SIS)
Carter
Uses bistance-wise backward selection to identify subsets of input features sufficient for the model to make its prediction above some confidence threshold.
et al.
2019)
RISE (Petsiuk al. 2018)
Computes the weighted sum of the model's output on masked versions of the input.
Identifies important image regions by segmenting the image into many overlapping regions and ranking the regions based on the sum of pixcl-atribution within each region.
XRAI
Kapishnikov al.
2019
Table 1: Summary of saliency methods discussed throughout the framework.
cards" (akin to "model cards" (Mitchell et al. 2019) and "datasheets" (Gebru et al. 2018) doc- ument individual methods and to better contextualize saliency results. Moreover, using a concrete example of ML-based radiology diagnostic systems, we show how downstream stakeholders can use our framework to weigh tradeoff's and choose task-appropriate method. Finally, we show how our framework identifies compelling opportunities for future work including exploring understudied dimensions and developing new metrics that target particular dimensions.
2 RELATED WORK: SALIENCY METHODS AND THEIR EVALUATIONS
Saliency methods (sometimes referred to as feature attribution methods) produce explanations for an ML model's 's output. Given an input, saliency methods compute an importance score for each input feature describing its influence on the model's 's output. Existing categorizations for saliency methods have focused on algorithmic properties (gradient or perturbation bratrbbttoon-busdd, wuth-attribution or gradient- only) Molnar 2019)). Our framework aims to categorize a broader range of important characteristics. We apply our framework to a variety of common saliency methods listed in Table !
Evaluations of saliency methods have primarily focused how accurately their results represent model behavior, often referred to as faithfulness. A growing body of work has identified failures of some methods such susceptibility to adversarial perturbations (Ghorbani al., 2019). lack of neuron discriminativity (Mahendran & Vedaldi 20166), and a predisposition towards input recov- ery NNe et al. 2018 Adebayo et a0.18) Other work has proposed proxy tests that measure different aspects of faithfulness. Adebayo et al. (2018) recommend model randomization and data label randomization tests, and Kindermans et al. (2019) test whether constant input shifts affect saliency results. [Samek et al. (2016) judge saliency methods by iteratively replacing features that have high importance values with random noise and measuring how much the output changes. While these tests quantitatively analyze saliency methods, Tomsett et al. (2020) found they can produce in- consistent rankings.
An alternate line of work has looked to break faithfulness down into measurable axioms. Sundarara- jan et al. (2017) propose implementation invariance (a saliency method should produce the same output functionally equivalent models). sensitivity (a saliency method should give importance to a feature if and only if changing it leads to a different output), and linearity (if a model is the composition of two sub-models, the saliency method's 's output for the model should be the weighted

---

Under review as a conference paper at ICLR 2022
Imput Sensitviity
Model Sensitvity
Algortthm
aood
but gave
Bood,
MODEL
Extract
food
Good, but food
Compute
posssomingg
Visualize
poisoniing
interpret
poisoninng.
SALIENCY METHOD
SALIENCY OUTPUT
SALIENCY MAP
HUMAN
INPUT
METHODOLOGY
PERCEP TIBILITY
Determiniem
Minimality
thperparemeter Dependence
Per Perceptual Corfespondence
Model Agnosticism
Semantic
Computational Fiticlency
Figure 1:
Our framework offers nine dimensions to characterize and compare saliency methods, which are grouped into three categories corresponding to different phases of the interpretation pro- cess: methodology describes how the saliency is calculated; sensitivity describes relationships be- tween input and output; and perceptibility relates to human interpretation.
sum of its outputs for each sub-model), SSundararajan et al. (2117) and Shrikumar et al. (2017) both posit saliency method should exhibit completeness -a a saliency method's s attributions should sum to the difference between the model's 's output on the input and the model's output on a neutral input. Shrikumar et al. (2017) and Montavon et al. (2018) claim saliency methods should output contin- uous function. MMntavon et al. (2018) also propose the axiom of selectivity a saliency method should distribute importance to features that have the greatest impact on the model's output. Finally, Kindermans et al. (2019) state that a saliency method should be invariant to constant transforms.
While axioms presented constraints that all saliency methods should attain, the axes in framework describe attributes of saliency methods that can be traded off to select the best method for different use cases. To situate saliency methods along each axis, we utilize existing tests (Kinder- mans et al. 2019; Adebayo et al.) 2018);; however. our framework surfaces the need for additional evaluations, since existing tests do not fully describe each axis and have not assessed all methods.
NINE DIMENSIONS TO CHARACTERIZE & COMPARE SALIENCY METHODS
To distill a language to characterize and compare saliency methods, we treat saliency methods as abstractions of the underlying model behavior. That is, to explain model behavior in human- understandable format (e.g., a feature attribution heatmap), we must necessarily abstract away some detail and concreteness. A saliency method, by design, is not precise reflection of model be- havior as a printout of the weights, but is much more iammannnterrpetabble. The idea of abstraction recurs in other parts of the machine learning pipeline. For example, "test accuracy" is abstrac- tion of model performance: it does not capture all aspects of model performance but is nevertheless convenient approximation. Abstractions selectively preserve information and can be combined to arrive at a more complete picture of the underlying phenomenon i.e., combining test accu- racy with AUROC more fully understand model performance. This framing reveals that a single saliency method cannot fully explain model behavior. Instead, it is necessary to understand different methods' strengths and limitations choose an appropriate method for the model, domain, and task.
Our framework defines nine dimensions that describe what information saliency methods abstract, and how this abstraction computed. We group these dimensions into three categories that map to different stages of the interpretability process (see Figure D Methodology covers the model, input, and saliency method, and dimensions under this category describe how the saliency method operates. Sensitivity goes one stage further, with dimensions that describe the relationship between the saliency output and the model or input. Perceptibility covers the final stages of the process, with dimensions that describe how users perceive the saliency output and any associated visualizations. These dimensions allow us to decompose faithfulness into aspects that can be reasoned about in- demonstrates how dimensions allow us to situate saliency
dividually. For instance, Figure 2 methods in relation to one another to understand their strengths, limitations, and differences. In the following subsections, we describe and provide illustrative example of each dimension.
2

---

Under review as a conference paper at ICLR 2022
Computational
-10o0s for 224x224-3 image
-10p-1s for 224-224-3 image
-10--2s 224-224 image
Efficiency: Speed of computation.
loss
SHAP
more
deternirisii (except if using baseline)
Determinism:
ron-determnnstic coaltions
deterministic
Consistent reproduction of results.
less
SHAP
VG
mirimality incentives; known to appear noisy
estimating global importance avoids gradient saturation and alleviates
regulorization model
Minimality:
Identification of minimal feature
less
VG
SHAP
input sensitivity depends baseling valua
input according constant from Kindermans
Input Sensitivity:
Reaction meaningful input changes.
less
IG
VG
Figure 2: Situating methods along dimensions of our framework reveals their multifaceted differ- ences (e.g., SHAP is more minimal but less tractable than VG). Understanding these differences can inform context-specifer decisions about how to choose and interpret methods appropriately. It also reveals gaps (e.g., SHAP has not been tested for input sensitivity) that can inform future work.
3.1 METHODOLOGY: HoW SALIENCY METHODS OPERATE
Determinism Some saliency techniques are non-deterministic, and running them with different random seeds can produce significantly different outputs. For instance, methods such as RISE or SHAP rely on mundomly-generate masks or coalitions. The output from a non-deterministic method represents only one example from a potentially high-variance distribution, and may be af- forded authority that does not account for its inherent uncertainty. Although we may never explicitly choose a method for non-determinism, we may nevertheless accept non-determinism in service of other dimensions such increased model agnosticism (e.g., RISE), perceptual correspondence (e.g.. SHAP), or minimality (e.g., MP).
Example. Consider a model assisting dermatologists diagnosing skin cancer. In Figure 3 a],,, we show that two runs non-deterministic method (LIME) on a melanoma prediction model result in different saliency maps. These variations may have significant consequences given small areas of the lesion or surrounding skin may be integral to the diagnosis. Visualizing and interpreting multiple runs together is time-consuming and confusing, but only looking one run could skew clinician's s judgment. Thus, in this case, it might make sense to prioritize method with deterministic outputs.
Hyperparameter Dependence Hyperparameter dependence captures how many hyperparameters or design decisions the user must specify to run a particular method, and how sensitive it is to them. If there are not sufficient resources or expertise to devote to hyperparameter optimization, simply using default values can lead to misleading results. Similarly, misleading confusing results might arise the hyperparameters were chosen based on a particular dataset but deployed a setting in which there is significant distribution shift. In situations like this, it makes sense to prioritize methods with low hyperparameter dependence. In other cases, where developers have dedicated appropriate time and resources to tuning, it might be preferable to use method dependent on hyperparameters (e.g., IG, SmoothGirad) because improves other dimensions (e.g.. minimality).
Example. IG computes feature importances by interpolating between "meaningless" baseline input and the actual input, accumulating the gradients at each step. A common practice to use a base- line value of all zeroes; however, in some cases, a zero baseline can be misleading and potentially harmful. Take model trained to predict bone fractures from x-ray images. Fractures often appear as black line in the bone, but, since they have the same pixel value as the baseline (zero), the IG result will indicate that they are not important. In Figure choice of baseline (black, white) has strong effect on the saliency map. Choosing an appropriate baseline requires time and a deep understanding of the data and method; if this is not available in particular application, is better to prioritize saliency method without important hyperparameters.
we show a similar example where the
Model Agnosticism Model agnostic methods (i.e.. SHAP) treat the underlying model as black- box, relying only on its input and output. On the other hand, model-dependent methods need ac- cess to various model internals, and may have specific requirements, such as differentiability (i.e.,
3

---

Under review as a conference paper at ICLR 2022
(a)
Hyperparameter Dependence
Agnosticism
Different
with Baselines
Applied Different Model Architactures
Output
Image
Black
Image
Random
Computational Efficiency
Input Sensitivity
Model Sensitivity
Run Sallency on ImageNet Image
GradCAM Adversarial Attack (from Jia 2020)
Smoothed VG Over Model Randomization
Original Predicted:
Attacked Predicted: water pig
GradCAM
GradCAM on
Fully Tralined
Fully Random
Original
Minimality
(h) Perceptusl Correspondence
Semantic Directness
Gradients Before and After Smoothing
Randomly Initialized Model
English Explanations of Salioncy Maps
Image
VG
VG SmoothGrad
Image
Vanilla Grad.
Integrated Grad.
LIME
SIS
important
learned by a surrogate model trained to mimic the original model's boundary on this input."
pixels necessary for the model to make its original decision with confidence.
Figure 3: Our framework defines nine dimensions of saliency methods, shown above with examples. Each dimension describes attribute of saliency methods that be reasoned about and traded off.
gradient-based methods) or a particular type of architecture (i.e., GradCAM requires a CNN). Model agnosticism priority for use cases when modularity is required, when ensembling models, when comparing across model architectures, or when the model internals are not accessible. However, if only considering specific architecture, one might opt for non-model-agnostic method like Grad- CAM that provides other benefits (i.e., increased model sensitivity).
Example. Saliency methods can be an informative tool to compare the behavior of different models on the same input. For direct comparison, the same saliency method should be used across models, and model agnostic method likely necessary. For example, in Figureßjc), we show how a model- agnostic method like LIME can produce explanations for CNN and random forest model, which would not be possible using a model-dependent method such as IG or GradCAM.
Computational Efficiency Different methods vary widely in their computational efficiency for example, perturbation-base methods (c.g., g., SIS, MP) are often more computationally intensive than simple gradient-basee methods (e.g.. VG and its variants). In time-sensitive, low-resource, or data-intensive setting (e.g., all of ImageNet), prioritizing computational simplicity (which may entail sacrificing along other axes) may necessary.
Example. Video modeling (i.e., action recognition) is computationally intensive because, unlike 3- dimensional image inputs (channels height x width), video inputs add fourth dimension (frames). Even though the original frame rate can be subsampled, the compute time required for Trame-wise saliency maps scales linearly with video length. As shown in Figure d), methods like Guided orders of magnitude faster than other methods like XRAI Thus, in the video modeling setting, choosing a computationally efficient saliency method may be necessary to interpret model behavior.
3.2 SENSITIVITY: RELATIONSHIPS BETWEEN SALIENCY OUTPUT AND THE MODEL/INPUT
Input Sensitivity Input sensitivity captures the concept that the result of a saliency method should reflect the model's sensitivity to transformations in input space. Prior work has attempted to quantify the input sensitivity of saliency methods by comparing results before and after applying a constant shift to the input (Kindermans et al. 2019) (i.e., positing that an input shift that does not change the model output should also not change the saliency map if the method is input sensitive). Other work defines the property such that if two inputs differ by only outputs, an senut-sensitive saliency method should assign a non-zes zero value to that feature. In some cases, might tradeoff input sensitivity for, e.g., minimality (as in SmoothGrad).
feature but result in different model
4

---

Under review as a conference paper at ICLR 2022
Example. Input sensitivity is particularly critical for applications that risk adversarial attacks when studying model behavior in the presence of these attacks. For example, Ady-watermark (Jia et al. 2020) adds watermarks to images that change the model's prediction. In Figure (e), we show an example from Jia et al. (2200) where the model correctly predicts salt shaker on the original image but, after the addition of watermark, predicts water jug. The GradCAM saliency is input sensitive and identifies the ad adversarial watermark as most important to the incorrect water jug prediction.
Model Sensitivity Model sensitivity is the concept that the result of a saliency method should reflect changes in the model parameters. Prior work has formalized notions of model sensitivity in different ways: for example, Adebayo et al. (2018) run a parameter randomization test (comparing the output of saliency method on the trained model and a aundolly-inttiaizeed model), and [Sun- dararajan et al. (2017) define an implementation invariance axiom (stating that a saliency method should produce consistent outputs on functionally equivalent models). In some cases, we might be okay with trading-off model sensitivity. For example, Guided BP is not model sensitive, but is input sensitive; if we cared more about examining changes across different inputs (as opposed to different models), we might choose it over a more model sensitive method.
Example. Prioritizing model sensitivity may be particularly important when comparing models. In particular, take the case where one compares model to its fine-tuned counterpart to understand the effect on the model's behavior. In this case, it is critical to have a method with model sensitivity; otherwise, the results might misleadingly indicate that the model has not changed. In Figure I we use VG, a model-sensitive method, showing that it is sensitive to cascading layer randomization.
3.3 PERCEPTIBILITY. HUMAN INTERPRETATION OF SALIENCY OUTPUT
Minimality Minimality refers to how many features are given a significant value in the saliency map. Methods that attribute importance to many input features, especially when applied to high- dimensional data, can produce results that are difficult to interpret. For example, several papers have observed that VG results appear noisy, sometimes giving almost every pixel in an image non-zero importance. On the other hand, methods like LIME and SIS directly incorporate minimality into their procedures using regularization and backward selection, respectively. Smoothing methods, such as SmoothGrad, also improve minimality by averaging over the results on slight perturbations of the input. At the same time, methods that do not prioritize minimality may have other benefits (e.g.. VG are model sensitive) and might be more appropriate depending on the context.
Example. Minimality is particularly important when interpreting high- -ighndimenioonll data. For exam- ple, when dealing with patient medical records, each record might contain hundreds or thousands of features. Without minimality, the resulting saliency values risk obscuring whatever signal is present and increasing the cognitive load required to interpret the results. In Figure 3lg), g), we compare results of saliency method that are not minimal (VG) before and after smoothing (SmoothGrad) is applied, demonstrating that minimality makes it significantly easier to glance at and interpret results.
Perceptual Correspondence Perceptual correspondence captures the idea that the perceived sig- nal in the saliency result should reflect the model's confidence. For example, if model is randomly guessing predictions, saliency values should appear correspondingly random. Perceptual correspon- dence is crucial for high-risk settings, where misleading signal could provide an unwarranted justification for essential decisions or lead users down incorrect paths. On the other hand, if saliency results are not interpreted individually for decision-making, but rather being used for larger-scale analyses of model behavior that will aggregate one-off artifacts (e.g.. Boggust et al. (2021) use ag- gregate metrics on saliency results across an entire dataset to measure human-AL alignment), we might prioritize low computational efficiency over perceptual correspondence.
Example. In Figure Kh), we show saliency maps on a randomly initialized model. While VG appears random, IG computed with a black baseline exhibits a perceptual signal because of the near- black regions in the images. A user might project meaning onto the saliency map, concluding that the model is picking up signals in the input when in fact it is entirely random. While we use an ImageNet image (Degg et al., 2009) to illustrate the concept, the lack of perceptual correspondence in IG would be particularly problematic in black and white medical images (i.e., x-rays, mammograms) where saliency maps might indiscriminately highlight white areas, providing unwarranted justification for further action treatments.
5

---

Under review as a conference paper at ICLR 2022
Sallency Card for Vanilla Gradients
Saliency Card for integrated Gradients
Method Details
VG: atterhound
Method Details
IG: cab (random baseline)
Overview: Gracient function
Overview: Integral the gradkent function
between the input and a neutral boseline.
input
Robust to saturated gradients.
Developers: University of Oxford.
Developers: fissearchiea Google
Reference: Simonyan al. "Deer Insste
Reference: Sundararajon, "Axomattic
Clasellication Models and Sallency Mara" (2014)
Comeodutional Networe for Visualiom
Computational Effeciency
Attriuttion for Deep Netwodds (2017)
Computational Dificiency
Methodology
Methodology
Determinigm..
Computational Efficiecc: -19e- 1s 274x724x3 image, Reshie500 Noddle G188
Computational Efficiency: 190 7% for 224x224x3 img. RasNan50, and Nvidia
erpprrameeerr Dependence
Determinisn: Determiiitti unless usingg
heperparamcter Dependencc
6100
ron-determisstie baseline
Hyperparamater Dependance: None.
Determiniam: Determiristic
Model Agricdiciam
Hyparparamater Dependance: Requires baseline
Model Agnosticiom
Model Agnosticlam: Requires difforemtiable
Model Agnostiatom: Requites differemiablc model with access to gracients.
model access to gracients.
Input Sensilivity
Input Seriitivity
Sensitivity
Sensitivity
Input Sanshtivity staya shift [1]
Model Senatthiity
Input Sensitivity: Reaction conatant [1] contingent on baseline value.
Model Donsitivity
Model Sensitivity: Output changes as
model parameties are nandomired [P].
125
Minimality
model parameters are randomized [al.
Model Sensitivity: Output changes as
Mininalty
Perceptibility
Perceptual Cromspondence
Perceptibility
Us Perceptual Carrespondnne"'
Minimality: mirimally incentives. May
Minimality: Alleviates noice caused by gradient saturation.
visually noisy [S] and manyy
features with nomoero importance (4).
Semartic Drectress
Percaptual Comeesandaecee: Not affected Featuree equal
Semantic Directness
gradient can mask signal from importane
Perceptual Cerrespondence: Saturated
Ciceemee The
mni Sethoal pari))
gradient the value will have zero importance Prediepostiion to inpu recovery
Kindemmar et "Thx dacen
featuress [S]
Lans (201))
Semantic Directness: magnitude
Bakaukn As sanaua
Bebena Mules" (2011)
change outpur given small channge
the input feature.
Cel mn Lana (2017)
gradients from baseline to the rnput..
Semantic Diractness:: Acsumu of
ESGRS Easttan Art.Mai Eatnloes'
"Shualitoata
Figure 4: Using our framework, individual methods can be documented in a standardized manner as "saliency cards," " informing users about tradeoffs and limitations, and facilitating rapid comparison.
Semantic Directness Saliency methods represent different aspects of model behavior. For exam- ple, the result from SIS represents minimal sets of pixels necessary for particular prediction; the result from LIME, on the other hand, represents the learned coefficients of a local surrogate model. Semantic directness refers to how straightforward it is for a particular user group to understand what the result of a saliency method represents. A semantically direct method may be particularly im- portant if users are unfamiliar with ML or saliency methods, in order to prevent misinterpretation of results. In other cases, we might trade it off for example, while SHAP (which defines feature importances as Shapley values, a method from game theory) may not be semantically direct to many users, it can improve perceptual correspondence, which might be more critical in high-risk contexts.
Example. Recently, a range of public-facing browser-based interpretability tools have heen devel- oped, such as the Language Interpretability Tool (LIT) (Tenney et al.) 2020). These tools might be helpful for data scientists or other users in a particular domain, who don't 't necessarily have formal ML expertise, to explore their data and a model's behavior on it. Within LIT, users can choose among several methods for viewing the saliency results of a model on selected sentences. In this case, a non-expert user might opt for more semantically direct method such as SIS (over, e.g., IG, SHAP, or LIME) that facilitates more intuitive and quick exploratory analyses without needing to understand the implications of a surrogate model or accumulated gradients.
4 FRAMEWORK USE CASES
4.1 DOCUMENTING SALIENCY METHODS WITH "SALIENCY CARDS"
By moving beyond singular notion of faithfulness, our framework gives us a rich, granular vocab- ulary to describe existing and future methods, and situate them in relation to one another. In doing so, our framework improves our understanding of different methods and can be used to document them in a standardized fashion. For example, in the same way "model cards" and "datasheets" pro- vide detailed, public-facing documentation about particular models or datasets, our dimensions can inform analogous "saliency card" as shown in Figure 4 Such cards step through the individual dimensions of the framework, providing a brief qualitative or quantitative description for each. By including an example of the method in action, well schematic figure that plots the method
6

---

Under review as a conference paper at ICLR 2022
with respect each dimension, these saliency cards can help better contextualize individual meth- ods in the broader landscape. Downstream users can rapidly compare cards to understand a method's strengths and limitations, as well as build intuition for the trade offss between different methods.
4.2 COMPARING AND CHOOSING SALIENCY METHODS IN A RADIOLOGY SETTING
The dimensions of our framework let us systematically evaluate, choose, and compose methods for different contexts. To illustrate this use case. we consider an application of ML-based diagnostic tools: interpreting chest x-rays. Radiologists interpret chest x-rays diagnose range of cardiopul- monary conditions, and automated analysis could potentially improve their workflows and extend expertise to low-resource settings. ML--based systems have gained substantial traction for this appli- cation, in part because of the availability of large, public datasets (Johnson et al. 2019; Irvin et al.) 2019). In practice, accompanying automated decisions with explanations critical, both for the systems to be usable and trusted by physicians (Tonekaboni et al. 2019) and to follow standards and laws for explainability (Smith 2020 European Commission 2018).
We evaluate our framework in the context of this case study through semi-structured interviews with 4 radiologists (see Section A.1 for interview details) to understand how our framework surfaces trade offs and helps radiologists compare different saliency methods. We find radiologists may be unfamiliar with the the way non-determinism manifests in saliency methods (i.e., LIME, SHAP), given that it is typically not encountered within other medical technologies. Non-deterministic sults could appear random and lead to speculation about whether an important feature would still be important if they reran the method. Given this, may want to prioritize determinism over other axes and choose a method, like VG, that is fully deterministic. While VG has less minimality than many -on-determiiistic methods, its noisiness may be preferable to the stochasticity of a non- deterministic method - background noise is a familiar concept in medical imaging and radiologists are used to accounting for it during interpretation.
Relatedly, minimality is often presented as a desirable attribute (Smilkov et al. 2017). Radiologists, however, are accustomed to using background noise attenuate measurement uncertainty in their work; thus, the absence of background noise could convey unwarranted certainty. Indeed, results in visualization research indicate that clean, minimalist visual representations can convey a misplaced sense of certitude about the underlying data (Kennedy et al. [016]] Kostelnick] 2008). This con- sideration also suggests another tradeoff between perceptual correspondence and minimality. If the saliency result misleadingly communicates a signal to which a radiologist attributes meaning, it could lead to a patient being misdiagnosed or wrongly treated. Given the high risk involved with incorrect decisions in medical context, we might actually opt for a less minimal method - despite the fact that its results may not be as glanceable- in order to increase perceptual correspondence.
Semantic directness is similarly presented as an beneficial characteristic for methods used with non- ML-experts (Carter et al. 2019). Radiologists, however, are accustomed to reading and interpreting scans produced via magnetic resonance imaging (MRI) technology despite having a limited tech- nical understanding of the physics underlying MRI technology. A similar ethos likely applies to saliency methods as well radiologiis might be comfortable receiving saliency results even with- out understanding the underlying mechanisms (c.g.. gradients, neural network architectures) as long as they know how use them.
The deployment process for a clinical decision model might also have implications on which di- mensions to prioritize. For example, IG's hyperparameter dependence could have significant conse- quences for x-ray data (e.g., the commonly used black baseline would lead to misleading results for x-ray images). This dependence might be more or less of an issue depending on the expertise of the person tuning the hyperparameters. If tuning was left to the software vendor, for example, we might prioritize a method that does not require consequential decisions like IG's baseline value.
Overall, the dimensions of our framework provide valuable structure for probing an application and reasoning about tradeoffs. While previously, we might have picked a method to use more arbitrarily, we can now systematically think about attributes to target. For example, non-determinism and perceptual correspondence are likely necessary for a radiology context, while minimality and semantic directness
not. Based on this, we can see how method like VG might be better suited than LIME for this particular task. Beyond specific methods, we can gain insight into general design implications, such as whether hyperparameters should be set beforehand or open user choice.
7

---

Under review as a conference paper at ICLR 2022
4.3 IDENTIFYING NEW RESEARCH DIRECTIONS
Our framework also allows us to systematically analyze the research landscape. We characterize 10 methods along each of our framework's dimensions (Table [AT] by drawing on existing work (i.e.. papers that present compare methods) and our experiments (i.c., rumning methods locally to compare their computational burden). In doing so, we identify several patterns and gaps in existing methods that suggest promising directions for future work to explore.
Some dimensions are rarely explored in existing work. For example, while it may be a crucial at- tribute for high-risk applications, few papers attempt to measure perceptual correspondence. This gap might be due to measurement difficully, since research characterizing and quantifying perceptual correspondence requires conducting user studies. Such inquiries could draw from work on human visual perception (Logothetis & Sheinberg, 1996) -for example, using eye-tracking to measure if and where in an image people fixate (Borys & Plechawska- -oooik 2017 Yarbus 2013) to designing standardized ways of studying when people read strong signal in saliency map visualiza- tions. A better understanding of perceptual correspondence could also inform the design of saliency visualizations that explicitly communicate limitations of the method and preemptively avoid imply- ing unwarranted signal. For instance, we can imagine a richer design space beyond static heatmaps where interaction is used to dynamically overlay multiple attributions (as in Olah et al. (2018) and more intuitively communicate uncertainty.
In other cases, existing work has already begun to devise evaluative metrics that map to dimensions in our framework. For example. [Adebayo et al. (2018) use a model parameter randomization test that helps gauge model sensitivity, and Kindermans et al. (2019) use test that compares saliency results before and after applying a constant shift that helps measure input sensitivity. However, these specific, narrow ways of measuring model and input sensitivity, and can produce inconsistent results (Tomsett et al.) 2020). By explicitly identifying these properties as separate dimensions, there is significant opportunity devise new or different ways to quantify them. For example, while Kindermans et al.] quantification of input sensitivity involves applying meaningless change to the input, we could imagine applying a meaningful change to the input and studying how the output changes in response (akin to algebraic visualization design by Kindlmann & Scheidegger (2014).
Once methods are well-characterizze across a particular dimension, other patterns might emerge for example, clustering within a dimension or combinations of dimensions. We noticed, for instance, that minimality is typically encouraged via regularization or smoothing terms, and thus the more minimal methods are typically more hyperparameter dependent. Are these dimensions inherently tied together, or is possible to achieve increased minimality in other ways? This is an example of research question that emerges from understanding gaps and clusters in the design space.
5 CONCLUSIONS AND FUTURE WORK
In this paper, we present framework for describing, communicating, and comparing saliency meth- ods. Unlike prior approaches, which have focused on a singular notion of "faithfulness" or how accurately saliency output reflects the underlying model, our framework offers nine dimensions to characterize and compare saliency methods that span different phases of the interpretation workflow. As result, our framework facilitates rich documentation of saliency methods (i.e., through saliency cards), and can help users understand trade offs and select methods that are appropriate for their data, model, and task. Finally, situating saliency methods along each dimension surfaces areas for future research including developing methods that prioritize specific attributes, designing tests to quantify each dimension, and proving (or disproving) entanglement between dimensions.
We intend our framework to start conversation around approaches to characterize and compare saliency methods, and be a "living artifact." Namely, as researchers develop new saliency methods or quantify particular dimensions (e.g., input sensitivity). we expect our framework to evolve. For instance, additional dimensions may be introduced to distinguish attributes or techniques proposed by novel methods, or new evaluative studies may reveal that existing dimensions are too broadly defined and warrant further decomposition into constituent, testable axes. Further work is also needed to provide formal (i.e., mathematical) treatment of our framework. Doing would help turn our framework from a purely descriptive instrument into a generative one as well where dimensions can be operationalized to systematically produce altogether new saliency methods.
8

---

Under review as a conference paper at ICLR 2022
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. arXiv preprint arXiv: 111.v:/81000:322,, 2018
Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Col- lisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. Guidelines for human-ai interaction. In Proceedings of the 2019 chi conference human factors in computing systems, pp. 1-13, 2019.
Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José MF Moura, and Peter Eckersley. Explainable machine learning in de- ployment. In Proceedings of the 2020 Conference Fairness, Accountability, and Transparency, pp. 648-657, 2020.
Angie Boggust, Benjamin Hoover, Arvind Satyanarayan, and Hendrik Strobelt. Shared interest: Large-scale visual analysis of model behavior by measuring human-ai alignment, 2021.
Magdalena Borys and Malgorzata Plechawska-oojee Eye-tracking metries in perception and vi- sual attention research. EJMT, 3:11-23, 2017.
Brandon Carter, Jonas Mueller, Siddhartha Jain, and David Gifford. What made you do this? under- standing black-box decisions with sufficient input subsets. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 567-576. PMLR, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 IFFE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Shuoyang Ding and Philipp Koehn. Evaluating saliency methods for neural language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pp. 5034-5052, 2021.
Dumitru Erhan, Yoshua Bengio. Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.
European Commission. Transparent information, communication and modalities for the exercise of the rights of the data subject, 2018. URL BRRL/https://ddp--iiit eu http/rttt1--ppppp.
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In Proceedings of the IEEE International Conference on Computer Vision, pp. p 3429-3437, 2017.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets. arXiv preprint arXiv: 180.:00000,0 2018.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3681-3688, 2019.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-llcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 590-597, 2019.
Xiaojun Jia, Xingxing Wei, Xiaochun Cao, and Xiaoguang Han. Adv-watermark: A novel wa- termark perturbation for adversarial examples. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 1579-1587, 2020.
Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lun- gren, Chih-ying Deng, Roger Mark, and Steven Horng. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):1-8, 2019.

---

Under review as a conference paper at ICLR 2022
Andrei Kapishnikov, Tolga Bolukbasi, Fernanda Viégas, and Michael Terry. Xrai: Better attributions through regions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4948 4955, 2019.
Helen Kennedy, Rosemary Lucy Hill, Giorgia Aiello, and William Allen. The work that visualisation conventions do. Information, Communication & Society, 19(6):715-735, 2016.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, pp. 267-280. Springer, 2019.
Gordon Kindlmann and Carlos Scheidegger. An algebraic process for visualization design. IEEE transactions on visualization and computer graphics, 20(12):21812219 2014.
Charles Kostelnick. The visual rhetoric of data displays: The conundrum of clarity. IEEE Transac- tions on Professional Communication, 51(1):116-13)) 2008.
Xiao-Hui Li, Yuhan Shi, Haoyang Li, Wei Bai, Caleb Chen Cao, and Lei Chen. An experimental study of quantitative evaluations on saliency methods. KDD '21, pp. 3200-3208, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781430383325. 3447548.3467144 348.148. URL /tL/tttt::////..rr///// 1145/14411334455511 ..1.777788..344444 3467148
doi: 10.1145/
Nikos Logothetis and David I. Sheinberg. Visual object recognition. Annual review of neuro- science, 19(1):577-621, 1996.
Scott Lundberg and Su-In Lee. A unified approach to interpreting model predictions. arXiv preprint arXiv:1705.07874, 2017.
Aravindh Mahendran and Andrea Vedaldi. Salient deconvolutional networks. In European Confer- on Computer Vision, pp. 120- 135. Springer, 2016.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pp. 220-229, 2019.
Christoph Molnar. Interpretable Machine Learning. 2019.
Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Methods for interpreting and un- derstanding deep neural networks. Digital Signal Processing, 73:1-15, 2018.
Weili Nie, Yang Zhang, and Ankit Patel. A theoretical explanation for perplexing behaviors of backpropagation-basee visualizations. In International Conference on Machine Learning, pp. 3809-3818, PMLR, 2018.
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 3(3),e10, 2018.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint (006.0780 2018.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should Trust You?": Explain- ing the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. Evaluating the visualization what a deep neural network has learned. IEEE transactions neural networks and learning systems, 28(11):260--267) 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
10

---

Under review as a conference paper at ICLR 2022
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, pp. 3145- 3153. PMLR. 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 2013.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint 1706:0706.038255 2017.
Andrew Smith. Using artificial intelligence and algorithms, Apr 2020.
URL
https:// httt:::/wwww.tcc.g////eeeeeeeee/
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 1412..4442.8 68066 2014.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution deep networks. In International Conference Machine Learning, pp. 3319-3328. PMLR, 2017.
Mukund Sundararajan, Jinhua Xu, Ankur Taly, Rory Sayres, and Amir Najmi. Exploring principled visualizations for deep network attributions. In IUI Workshops, volume 4, 2019.
Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, al. The language inter- pretability tool: Extensible, interactive visualizations and analysis for nlp models. arXiv preprint arXiv:2008.0 122 2020.
Richard Tomsett, Dan Harborne, Supriyo Chakraborty, Prudhvi Gurram, and Alun Preece. Sanity checks for saliency metrics. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 6021-6029, 2020.
Sana Tonekaboni, Shalmali Joshi, Melissa D McCradden, and Anna Goldenberg. What clinicians want: contextualizing explainable machine learning for clinical end use. In Machine learning for healthcare conference, pp. 359-380. PMLR, 2019.
Alfred L Yarbus. Eye movements and vision. Springer, 2013.

---

A APPENDIX
Determinism
Hyperparameter Dependence
Model Agnosti- cism
Computational Efficiency
Input Sensitivity
Model Sensitivity
Minimality
Perceptual Corre- spondence
Semantic Direciness
VG
Determinissiic.
Nore.
Kequires eniiable model and access gradients.
~o 10e22 secunds for 224x224x3 image.
Saliency results stay constant after constant ingut shift.
Saliescy results change us model purameters are raadomized.
Retalts appear seisy gradients for impoctunt features ure suturated and normalized within example.
Saturated gradicnts cun mask signal from important features.
magninde of change the uuipui given a small change in un inpui feature.
Results are more mini- mui thin VG since only high gra- diens and inpar values appear important.
Input Gradient
Deterministic:
Requines differ- model and access to gradients.
10e-2 seconds 224x224x3 unage.
Saliency results change stani imput shilf.
Saliency resulls change model parameters are randomized.
Predispociiin to input rocovery: susceptible gradieni saturation.
ingur value weighted by the gradient
Under review as conference paper at ICLR 2022
Predispostion input recovery (can be exae- erbated by choice of baseline). Addresses gradient saturation problems..
Deterniniistic un- less using nundom beseline.
Boselise value, number of sieps estimate the integral.
Requires diffes- emiiable model and gralients.
~ 1000e-l seconds for 224x224xx inage.
Reaction consiant shift test dependent haseline value.
Saliesey rosults change modlll purameters rendomized.
Can alleviate noisueess from gradiens satu: by estimatiing global impoctusces.
The accumulazed gradient between the baseline value and and feature value.
IG
Interpolation to upssmple to the original feature space. Choice of convolu- fional layer Olyicallly last convolutional layer?.
GradCAM
Deterministic.
Kequires differes- cable model, access gradienss, and convolutional layers.
~ 10e- seconds for 22/xx26xx3 image.
Saliesey rosults charge modell purameterrs randomized.
Interpolating back to original input size pro- duces low resolution result.
The positive atributions of the gradiemt weighted feature mape from the convalutional layer.
SmoothGrad
Non-dzterminiitic perturiations.
Guussian noise param- evers; number of runs average
Can he applied to saliency method.
Adds 10x time increase for 2222222 image.
Reaction constant shift test inherited from underlying salieney method.
parameetes randomiza- tion inhereted from underlying saliency method.
Reaction to
Minimallty escouraged by averaging over multiple results.
average many nins of another saliency method on penarbed versions the input.
Salieney resuilts do not change when higher layer weights are randomized, and only change when lower layer weights randomized.
output another gradient bused saliency method rrypecally VG), only considering puths through the network with posntive gradicnts.
Guided Backprup
Deterministic.
None.
Requires emiibble model and gradients.
-- 10e-2 seconds for 224x224xc3 image.
Saliency rosults stay constaat after a comskani input shift.
Minimality encoeraged through ramoving negative gradients.
Predisposition to inper recovery.
posntiely contribuz- ing features leurned surrogate model trained mimic the original model's local decision boundacy for a pasticulur input.
Lineur surrogate mod- els search over; input perturbasion lineur model parumetesization.
Non-daterministic pesturbations.
No requirements on the model access internals needed.
~ 10el secords for 224x224 image.
Minimallit excouraged through regulatization in local linear model.
LIME
GradSHAPP: 10e-
Fealures with int- pact on model given zero importance in resuli. Features that impuct equally reasive agual attribution.
Feanure replocement values; linear model parameterisation regu- larization parameter.
No requirements on the model neoded.
seconds for a 224x224x3 im-
Minimality encouriged through regulrization in local model.
The impact of each input feature on the defined by Shupley
Nan deserministic coalition sampling.
SHAP
KernelSHAA.:
10e0 seconds for 228x224x3 image.
Minimality escouraged through regulasization on the metz. predictor that leams explanatory rules.
Muskks pixels with non deterministic perturbations.
Regularization para-- eter; nouse and blur parameters.
No requirements on the model access internals needed.
The minimal region the image that would canse the langes change removedd.
MP
Alguriihe de- seribeed in Carrer
The minimum set of pixels necassary for the model to produce the curput with probobility above given threshuld.
Determanisticlyy produces of explanations per input.
Featur replacement values: model confi- dence threshold.
requirements the model access to internols neaceed.
Selects manimal of features sufficiens to meet the confidance threshold.
SBS
ct 2009 pro-
hilsrwaly slow on 224x224 images.
The weighted sum of the model's confidence of
RISE
Non-daterminiatt generation
Masking value, misk gesemtion parameters)
No requirements os model internals needed,
100- socandk for
224x2244 image
Mask upsampling encourages comtinous salient regions.
masked the input inputs include that feature
Requires with inpurs whoss
Region-impprtinn
XRAI
Determinestic
Segmentation method. intribution method (pu per pruposed IG with black and white lines)
featunes can be mesningrally clui- tagether teg.
224x204k3 image)
10el seconds for
importtnne:
Outputs
region-lovel
bean shows more human interpretable than pixel-importann (Sundrrrrgan 2019)
The regions with the largest sum of leature aniibutionn
image pisels))
Table A1: We situate 10 saliency methods along the nine dimensions of our framework, finding that they capture meaningful and distinet attributes about the methods. Some dimensions are characterized via our own experiments (e.g., computational efficiency) and others draw from tests described in existing work (e.g.. model and input sensitivity). In this way, our framework lets us systematically analyze the research landscape to understand gaps and opportunities for future work.

---

Under review as a conference paper at ICLR 2022
A.1 RADIOLOGIST INTERVIEW DETAILS
To evaluate the effectiveness our framework, we interviewed 4 radiologists and report the results in Scction 4.2 We recruited radiologists via a colleague who is practicing radiologist at a hospital. We did not require radiologists to have any familiarity or experience with machine learning. Each interview lasted approximately minutes. During the interview we discussed dimensions our framework and gave clinical examples on CheXpert (Irvin al. 2019) chest x-rays (see Figure A5 We asked radiologists open- open-ended geeneended questions with the goal of understanding:
How do radiologists think about individual dimensions of the framework?
How do radiologists prioritize and trade off different dimensions?
How well does the framework enable effective communication about saliency methods be- tween the radiologists and ourselves?
Hyperparameter Dependence
Perceptual Correspondence
Reliance on human specified parameters.
Visual alignment with numerical meaning.
Cheel K-aay
Vedia Gradleet
Integeael Beaderk
Panndags Attectatl 50.7%
Unimmine Model
Figure A5: Chest x-ray examples shown to radiologists in our interviews.
13


Reviews:
[{"review_id": "xcfiZxP-pFv", "reviewer": "Reviewer_JNYz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\nThe underlying idea of framing saliency methods as abstractions that provide an approximate view of a model’s behavior is insightful. This viewpoint sidesteps the philosophical arguments about which measure is best, and instead acknowledges that different aspects of saliency methods, measured in different ways, are useful for characterizing the algorithms. One can argue about the specifics of the framework, i.e. whether the three categories are the best ones, or the nine dimensions similarly, but the underlying concept remains sound and produces a very useful analysis.\n\nThe paper format of introducing each dimension, providing its rationale, and a concrete example with a result is very effective at clearly communicating the definition and utility of the dimensions. Overall the paper is very clearly written.\n\nThe saliency cards, Fig. 4, are an excellent snapshot of each saliency algorithm. These types of cards have proved to be useful in other domains, as the authors state, and should be quite useful for user-facing algorithms such as saliency.\nThe analysis summary in table 1A is a compelling example of how the framework can comparatively reveal strengths, weaknesses and gaps in particular algorithms. \n\nIt is more than a survey paper, through its careful definition of the nine dimensions, many of which are new to this work. The concepts of perceptual correspondence and semantic directness seem particularly unique; the latter attempts to quantify the inherently squishy notion of the ease of user understanding of a method’s results.\n\nWeaknesses\nAll of the highlighted saliency methods are from 2017 or earlier, except one from 2019. While it is reasonable to focus on those that have established themselves over time as impactful, it would be helpful to incorporate a few more recent methods to ensure that they still fit into the framework.\n\nWhile the paper should be quite useful for researchers and practitioners of saliency algorithms, it does not present novel work to build upon. Some may find this insufficient for ICLR.\n", "summary_of_the_paper": "The paper describes a new framework for characterizing and analyzing saliency methods. Nine dimensions are defined, grouped into three high-level categories, with some form of quantification in each dimension to enable comparative measures. The dimensions are used to formulate saliency scorecards, concise 1-page summaries of the saliency attributes of a particular algorithm that should help a user determine which algorithm is best for their needs. A set of ten recent algorithms are analyzed and compared using the framework, revealing that some have gaps in their assessments that could be address with future work. Compelling, illustrative examples are used throughout to convey understanding of the dimensions and the differences between them.", "summary_of_the_review": "Papers like this one tend to be rejected from highly selective conferences, but they can be very useful and impactful for researchers and practitioners alike in popular areas, such as saliency algorithms. This paper is thoughtful, well positioned with respect to previous attempts to characterize saliency algorithms, and should be an excellent resource for the saliency research community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "HbPrKmST1TT", "reviewer": "Reviewer_DxGz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is entirely a survey/review of existing methods; it does not propose any new XAI algorithms. However, I found the proposed dimensions and comparison of the saliency methods along these dimensions very interesting and informative. I could see this framework being useful to researchers in XAI as well as end users. The paper is also very well written and clear.\n\nThe main weakness of the paper in my opinion are:\n\n(1) The methodology is not entirely transparent -- it's not clear how the models are ranked on these dimensions or how a \"score\" would be computed for a given dimension. The authors note that some dimensions like \"perceptual correspondence\" are difficult to measure and may require large-scale user studies; other dimensions may be measured in multiple different ways and different methods could give different ranking results. It's not clear how this would be handled, but it seems to undermine the usefulness of this framework.\n\n(2) Since the proposed framework hasn't actually been implemented, there is no evaluation of how it works or whether it helps end users choose better models and make better decisions.", "summary_of_the_paper": "This paper proposes a framework to evaluate saliency-based explanations for neural network decisions. The framework evaluates each method along a number of different dimensions (for example, reporting whether the saliency map output is fixed or stochastic, or the degree to which the output depends on hyperparameters). The proposed framework is not actually implemented or tested in this paper.", "summary_of_the_review": "This paper proposes a useful framework for evaluating and comparing saliency-based explanations for neural network results. However, it is entirely theoretical/speculative -- the framework has not actually been implemented or tested.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Am-VU0KZJAh", "reviewer": "Reviewer_2CJF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The article is well written and provides a clear overview of some of the relevant works. It would have been interesting for the literature review to discuss in more detail the difference between the chosen approaches, as they calculate notably different values. \n\nThis leads to the main question from reading this article: Is it the case that the proposed taxonomy is an improvement on the state of affairs, by providing a clearer view of the pros and cons of competing approaches, or is it instead likely to obfuscate further the fact that most of those approaches compute distinctly different quantities. The paper does not provide actual evidence (for example, use case or user feedback) of this added value. \n\nSimilarly, saliency is defined as an abstraction in the introduction, but an abstraction to what is not clearly discussed, nor is the fact that there may be a disconnection between the expectations of the user and the actual quantities being calculated by different methods. ", "summary_of_the_paper": "This article proposes a review of \"saliency\" methods in deep neural networks and rates them according to 9 dimensions: Computational efficiency, determinism, hyperparameters, model agnosticism, input sensitivity, minimality, perceptual correspondence and semantic directness. The author's claim is that this new taxonomy provides a better way for choosing between saliency approaches for the end-user. \n\nNote: What is meant by saliency in this paper is methods, such as Grad-CAM to determine which inputs were more prevalent in a network's decision (the word has different meanings in other sub-fields). ", "summary_of_the_review": "\nIn summary, this article provides 1) a review of so-called saliency approaches for neural networks and 2) a new taxonomy for rating the properties of those approaches. \n- The article would benefit from a more in-depth discussion of the fundamental differences between the approaches considered and what it means for the term \"saliency\". \n- The article would benefit from some sort of evidence that the produced taxonomy is of value to the end users. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviews end up split. The most positive reviewer notes that the paper may be useful broadly to researchers and practitioners. That is the main promise of the work. However as another reviewer points out, the authors fall short of convincingly explaining how the said practitioners and researchers would benefit from the work. And, also as noted in a review, the paper does not quite go deep enough into the discussion of what saliency means for different methods analyzed. 
Contrary to the positive reviewers comment, I do not think that a paper must provide "novel work to built upon", but for a paper that doesn't, there is a significant threshold on how convincing it must be regarding the potential impact of the work. I think the paper in its current state, even after the rebuttal/revision, does not meet this threshold.
Decision: Reject