Path Integrals for the Attribution of Model Uncertainties

Abstract: Understanding model uncertainties is of key importance in Bayesian machine learning applications. This often requires to meaningfully attribute a model's predictive uncertainty to its input features, however, popular attribution methods are primarily targeted at model scores for classification and regression tasks. Thus, in order to explain uncertainties, state-of-the-art alternatives commonly procure counterfactual feature vectors associated with low uncertainty and proceed by making direct comparisons. Here, we present a novel algorithm for uncertainty attribution in differentiable models, via path integrals which leverage in-distribution curves connecting feature vectors to counterfactual counterparts. We validate our method on benchmark image data sets with varying resolution, and demonstrate that (i) it produces meaningful attributions that significantly simplify interpretability over the existing alternatives and (ii) retains desirable properties from popular attribution methods.

Reviews and author responses:

 Hi authors, thanks for reading the reviewer comments and taking them into account. About the metrics question - this seems pretty important because multiple reviewers brought it up.

The approach used in the XRAI paper seems pretty reasonable. There are different variations on this approach though, making it a bit hard to choose, but something like this could definitely be useful in your case. Several papers follow the basic idea of using the explanation to rank features, then deleting subsets to see how the model's predictions change.

For example, regarding the choice of how to "delete" features:
- The XRAI paper blurs portions of the image [1]
- The RISE paper zeros out parts of the image [2]
- The TreeSHAP paper samples alternative values for missing features (among other things) [3]
- The REAL-X paper uses an evaluator model trained with missingness to make predictions with subsets of features [4]
- The ROAR paper retrains the model when giving access only to the selected features [5]

After choosing how to delete features, most papers tend to analyze the predictions themselves, but I suppose you would want to look at the predictive entropy.

Next, you could generate a score for each method by:
- Measuring how quickly the entropy goes up when you include features in order of highest-to-lowest scores (e.g. you can trace out a curve for each sample $x$, calculate the AUC, then average this across many samples)
- Measure how slowly the entropy goes up when you include features in order of lowest-to-highest scores (the RISE paper does the AUC calculation in both directions [2])
- Measure the correlation/rank correlation between cumulative scores and predictive entropy (like this paper does with the sensitivity-n metric [6])

Some version of this could definitely work in your case, you might just need to think about how to justify a setup that makes the most sense in this context.

[1] Kapishnikov, Andrei, et al. “XRAI: Better Attributions Through Regions” (2019)

[2] Petsiuk et al., "RISE: Randomized Input Sampling for Explanation of Black-box Models" (2018)

[3] Lundberg et al., "From Local Explanations to Global Understanding with Explainable AI for Trees" (2020)

[4] Jethani et al., "Have We Learned to Explain?: How Interpretability Methods Can Learn to Encode Predictions in their Interpretations" (2021)

[5] Hooker et al., "A Benchmark for Interpretability Methods in Deep Neural Networks" (2018)

[6] Ancona et al., "Towards better understanding of gradient-based attribution methods for Deep Neural Networks" (2018)

 We would like to thank the reviewer for a detailed analysis of our research work and suggested improvements. We do agree that a quantitative metric would greatly aid in rigorous comparisons between different attribution methods.
We've performed initial tests with a small modification of the 'performance information curves' metric described in the following reference and obtained promising results. We thank the reviewer for the kind offer to recommend some appropriate metrics and would greatly appreciate it if they could either contact us to discuss it further or provide their recommendations here.

[1] Kapishnikov, Andrei, et al. “XRAI: Better Attributions Through Regions.” 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 4948–4957.

 We would like to thank all reviewers for the time they took to analyse our work. The concerns raised have been a valuable source of feedback for us, and we do acknowledge that our work would benefit from a more quantitative evaluation of saliency maps, and robustness testing against the implementation of variational autoencoder and the choice of hyperparameters. We have performed some initial tests towards quantitative comparison of different attribution methods and obtained promising results. However, performing more rigorous experiments and evaluations needs more work.

In this paper, the authors consider the question of how to explain model uncertainty in terms of the input features. Whereas prior work (e.g., CLUE) uses a counterfactual approach, the authors develop an attribution-based approach (assigning a score to each feature). The particular method is built on Integrated Gradients (IG): a baseline is identified that resembles the input but has no prediction uncertainty, and a non-straightline path is followed by traversing a VAE's latent space.

The experiments suggest that the proposed method (IG-H) has some advantages relative to CLUE, as well as simple adaptations of IG/LIME/SHAP to uncertainty attribution. Overall, the method appears to be sound and reasonable, it seems quite efficient and the results look positive - modulo some qualms I have about the experiments (particularly the lack of metrics). The problem area is also an interesting and important one. Below, I'll delve into some possible areas of improvement.

**The role of Bayesian model uncertainty.** The authors claim, beginning with the first sentence of the paper, that this method is designed for Bayesian models. However, even non-Bayesian models can have predictive uncertainty, particularly classification models (it may not be well-calibrated, but that's a separate issue). I wonder, then, why the presentation is focused on the Bayesian setting. The only advantage I can see is that it permits a distinction between aleatoric and epistemic uncertainty and the generation of separate entropy/aleatoric/epistemic explanations; this is nice in theory, but the authors don't really make the case that this is useful - or at least it doesn't appear useful in the experiments. This may be worth rethinking, because it's not advantageous to the authors to restrict their method's application, and it's misleading to readers to imply that non-Bayesian models don't permit the analysis of predictive uncertainty.

**Criticism of counterfactual uncertainty explanations.** In the introduction, one of the main criticisms of counterfactual approaches ("most importantly") is that they don't satisfy properties like completeness and implementation invariance. That's an odd criticism, because those properties don't even mean anything outside the context of feature attribution explanations: how, for example, could a method for generating counterfactuals satisfy something resembling completeness? With this in mind, is the real criticism then perhaps that counterfactual explanations lack properties like the IG or SHAP axioms to imply a unique, principled approach? That point would make more sense to me. Also, aren't the authors subjecting themselves to any issues inherent to counterfactual explanations *by using counterfactuals as part of their method?* Anyway, the point is, this came across as a facile argument and it would help to think this through more carefully.

**A multitude of ways to do uncertainty explanation.** The first paragraph of section 2.1 is getting at an interesting point: once we've decided to explain predictive uncertainty rather than the prediction itself, we find ourselves in the situation of being able to apply virtually any existing explanation method by just swapping out the quantity to be explained. I believe this is one of the key points made in Covert et al. (2020), and it might be worth saying that more clearly. The authors seem to realize this, because they adapt versions of IG, LIME and SHAP to do uncertainty explanation in their experiments. In my view it would be better for readers if the authors described the situation more clearly (the fact that we can adapt any explanation method, either removal-based or gradient-based), discuss the reasons why we might prefer gradient-based over removal-based (as they appear to think this is the case) as well as what modifications are necessary (e.g., having a baseline that encodes low uncertainty rather than high uncertainty) rather than just jumping into their method with these questions left unexamined.

About the method and experiments:

**Arbitrariness of counterfactuals.** There are many aspects of the method that could lead to different counterfactuals and hence different explanations. This is not a desirable property in an explanation, and the experiments leave these potential points of sensitivity virtually unexamined. For example, what if changing the optimizer or learning rate in algorithm 1 yields a very different result because of non-convexity? What if we had a different VAE? What if changing the distance metric, even by a constant multiplicative factor, yields a different counterfactual? The results may be very sensitive to the explanation algorithm's implementation; ideally this wouldn't be the case (none of the other baselines seem to have this unwanted property), but given that it is, could the authors do some sensitivity analysis to see how critical these things are? Or at least acknowledge this somewhere in the main text?

**Lack of metrics.** The experiments are purely qualitative and pre-suppose that we (the readers) know the ground truth for what contributes to the uncertainty in each image. That's maybe reasonable for figure 5, but probably not for figures 6 or 7. How, for example, are we supposed to know why the model is uncertainty about these FashionMNIST examples? I'm pretty opposed to accepting any model explanation paper that doesn't verify its results in a quantitative manner, so I would strongly encourage the authors to come up with some more rigorous metrics during the revision period. If they want to discuss this, I'd be happy to make some recommendations. 

**How much work is done by the counterfactual vs. IG?** I'm concerned that IG-H works because of the counterfactual generation step and that the IG component is unnecessary. When determining a counterfactual, we find an input similar to $x$ that is likely to differ in a small number of dimensions (particularly if we use $\ell_1$ distance for $d$). For features that are unchanged, aren't we guaranteed to then see zero or near-zero attributions? That suggests that the counterfactual step plays a critical role in making the explanation sparse and determining which features are eligible for non-zero attributions. 

With that in mind, how good would the explanation be if we just looked at $|\psi(z^0)_i - x_i|$ for each feature/pixel $x_i$? Or how about if we took $\psi(z^0)$ and used it as a baseline in IG with the straight line path? Or what if we used it as a baseline for SHAP? I would be very curious to see if any of these work. Unfortunately, it's going to be pretty hard to tell without any metrics, hence the request above.

**Smaller issues in the experiments.** In figure 5, the "proposed" column does not appear to match the "entropy" column, what's going on with that? "Vanilla IG" is a confusing name for the version of IG adapted to explain predictive uncertainty, is there a better name for this? Describing SHAP (and to some extent LIME/IG) as making an error by identifying regions that decrease uncertainty is a bit of a misunderstanding of what's going on: SHAP, for example, has a baseline (with all the inputs removed) with high predictive uncertainty, so it's natural that some features decrease the predictive uncertainty. Describing LIME as sensitive to the tuning of the segmentation algorithm is true, but you didn't have to tune it to give super pixels quite this coarse.

A couple smaller issues:

**Gaussian smoothing.** The use of Gaussian smoothing is a bit unusual, why is that necessary? It's a bit strange to describe the IG properties as very important and then apply an operation that may distort them. What would the explanations look like without smoothing? Did you apply the smoothing to all explanations in your results figures, or just IG-H?

**Confusing presentation of IG-H.** The equations that describe IG-H look like they may be unnecessarily complicated. For example, you could have just substituted $H$ or $H_a$ into the IG equation in the place of $f$ rather than applying product rule without saying so to arrive at the version shown in the text (see the two equations below eq. 3). Why is this version necessary to present, is it relevant to how you calculate IG-H in practice? Or do you calculate the predictive entropy and differentiate that with autograd? Overall, this paper considers an important an interesting problem and designs a pretty reasonable method. I have some qualms with the method and experiments (described above), but the most important one is the lack of metrics. If that were fixed, or if the authors made other improvements based on the issues described above, I would be inclined to raise my score.

In this paper, the authors propose an attribution method inspired by Integrated Gradients (IG), with the goal to explain the entropy using an in-distribution path (as opposed to vanilla IG that uses a linear path in the original domain). The authors show that the proposed method yields saliency maps that look more sensible for a few datasets. Feature attributions are an important framework within interpretable machine learning, and IG a mathematically grounded method to generate such attribution maps. Therefore, extensions to it are an interesting area to explore. The paper does a good job at establishing the problem and going over the two central ideas — use of entropy as the explanation target (as opposed to say logits), and use of in-distribution paths. However, I had a few concerns:

1. The ideas presented in the paper are fairly incremental. Namely, changing the explanation target from p_i to E[H(p_i)], and using a different path. The former is quite straightforward, and for latter there is existing follow-up work to IG considering other paths. See for example Blur IG [1] and Guided IG[2].

2. The evaluation is severely lacking. The paper shows some results on different datasets, but it’s mostly handpicked images, with justifications like “we notice that our attributions are comparatively neat, interpretable and always restricted to facial features”. The field of explainable AI has been known to just share such qualitative findings but these are prone to experimenter bias and insufficient to convince readers.

3. The combination of preceding two points means that there are missing comparisons with latest techniques making the claims of the paper appear weaker. Even for vanilla IG, just using two baselines (black+white) followed by averaging shows improvements and is missing here.

[1] Xu, Shawn, Subhashini Venugopalan, and Mukund Sundararajan. "Attribution in scale and space." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[2] Kapishnikov, Andrei, et al. "Guided Integrated Gradients: An Adaptive Path Method for Removing Noise." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
 Overall, while I think the problem domain is relevant, the paper’s contributions are incremental, and the evaluation fails to be convincing for me. I am not inclined for the paper to be accepted. 

The authors propose an extension of the existing path integrated gradients (IG) approach for the attribution of uncertainties of Bayesian models for image classification tasks. IG constructs straight paths between a fiducial image with high predictive entropy and the given image and integrates the uncertainty contributions along this path. The authors argue that this method suffers from the limitation that the paths may pass through images outside the data mainfold, i.p., because the proper choice of fiducial image is unclear and thus often defaults to a black image. To alleviate these issues, the authors assume the existence of a variational autoencoder model trained on the dataset with which they aim to generate in-distribution paths: they define the path as the decoder image of a straight path in latent space, the endpoints of which are defined by optimization problems keeping both of them in-distribution (and the fiducial image close to a image of the same class with low predictive entropy). In experiments on MNIST, Fashion-MNIST, and CelebA the authors compare their method to existing approaches and argue that their method yields easier interpretable results. The authors propose an extension of the existing IG approach by a novel method for constructing in-distribution paths. I think this approach is interesting and the basic idea well-justified. The authors provide extensive experimental evaluation. The results are purely qualitative in nature. Visually, the proposed method seems to be easier to interpret than existing approaches at the expense of relying on a variational autoencoder model trained on the dataset under consideration. However, it is hard for me to judge the significance of the results. The submission is mostly clearly written and well organized. 

The submission could further be strengthened:

- The submission lacks a detailed discussion of the final part $\psi(z) \rightarrow x$ of the constructed path. Why can we assume that the part $\psi(z) \rightarrow x$ is insignificant w.r.t. changes in predictive entropy? If this is the case, why don't we just define the endpoint to be $\phi_\mu(x)$ instead of $\psi(z)$ in the first place?
- The authors do not discuss the influence of the assumed existence of a VAE on the applicability of their work in comparison with existing methods.
- The authors do not discuss how to choose the hyperparameter $\lambda$ defining their optimization problem. Are their results sensitive to $\lambda$? How is $\lambda$ tuned?
- I feel that Sec. 3.3. lacks depth, i.p. because the authors stress that their method "respects desirable properties of completeness, sensitivity(b), and implementation invariance". I would like to see definitions and more detailed discussions of those terms as they seem to be central to the submission.
- The equations below Eq. 3 define the same symbol $\Delta_i(\alpha)$ with different right hand sides. The proposed method is an interesting extension of prior work. Visually, the results indicate an improvement over existing approaches. However, the significance of these results is hard for me to judge. As the submission lacks detailed discussions of key points, I lean towards rejection. However, I will raise my score if the authors improve upon the points I mentioned in the main review.

The authors propose a method to interpret model uncertainty for individual input examples. That is, it attempts to identify which input features (e.g. pixels of an image) are contributing to prediction uncertainty (measured as entropy in the multi-class prediction probabilities). To do this, for some query input, the proposed method applies integrated gradients between the input and a "fiducial" (i.e. high certainty) reference, which serves as a version of the input which is similar, but the network is very certain of its prediction class (the same prediction class as the query input). To find this fiducial reference, the method uses a pretrained variational autoencoder to find an input in the latent embedding space of high prediction certainty, that also lies closely on the latent embedding manifold. Integrated gradient interpolation is done linearly in this latent space. The authors show through several examples that the explanations for uncertainty are reasonable, and compare them to other methods like CLUE, LIME, and SHAP. ### Lacking global comparison metrics
The individual examples shown are both diverse and nice-looking, but I think the paper's main weakness is a lack of a global metric to summarize the improvements seen by using this method compared to benchmarks (e.g. vanilla IG, CLUE, LIME, SHAP). Given the size of these datasets, seeing individual examples is important, but global metrics are also important to convince readers that it works properly (and not just on a few cherry-picked examples).

### Limitations of using a VAE
The use of a VAE is unfortunately a limiting factor, as it does require training a separate neural network. Since the proposed method depends so heavily on this latent space learned by the VAE, it is extremely important that the VAE is trained well. This is a bit unfortunate because VAEs can be a bit tricky to train, given their competing loss functions. I think this is an important limitation to keep in mind, and it might be worth quantifying how much the performance of the proposed method changes by using different VAE architectures, latent space dimensions, etc (or different autoencoder types).

Sanyal and Ren, 2021 (Discretized Integrated Gradients for Explaining Language Models) proposed (somewhat recently) improving IG explanations using embeddings (by ensuring that the integration path stays within the training distribution). Many lessons from that paper are also seen here. This may be a paper that should be cited (if not cotemporaneous).

Sanyal and Ren, 2021 suggest some alternative methods to identifying an in-distribution integration path, also based on embeddings. For example, you could perform anchor search using embeddings that come from the last non-output layer of the classification model. This might be a better reflection of the latent space _from the point of view of the classifier we are trying to query_, and also bypasses the need to retrain a VAE. The downside is that it depends more on the model architecture.

### Theoretical justifications
A more minor point: the theoretical justification of uncertainty attributions (section 2) is appreciated, but it was not clear to me how the derivation of $H_{e}$ is important for the method. Especially so early in the manuscript, I found it to be a bit distracting. I would suggest putting most of these theoretical justifications in the supplement/appendix.

Additionally, some of the math could benefit from more detailed proofs, particularly the epistemic - aleatoric decomposition of $H$, and the derivation of IG with uncertainty. The main contributions of this paper seem to be: 1) applying integrated gradients to explaining local model uncertainties (by explaining the difference between a query input and an analogous high-certainty reference); and 2) finding in-distribution integration paths by using a VAE. The idea is interesting, and the example results in the paper are encouraging. However, the paper lacks global metrics that summarize improvements (or lack thereof) across entire datasets (or large random samples of datasets). If the authors can include convincing global metrics of improvement, then I would support its acceptance.

Meta-review: The paper explores a method to identify features in an input that can explain uncertainties in the model prediction. The proposed approach is similar to Integrated Gradients (IG),with a different explanation target and integration path. Overall, the idea seems fairly incremental and the experimental evaluation is lacking and does not sufficiently demonstrate the advantages of the proposed approach. Evaluation metrics could be improved (see suggestions by reviewer n3ei) to strengthen the paper.

Decision: Reject