Path Integrals for the Attribution of Model Uncertainties

1 INTRODUCTION . Estimating and understanding model uncertainties is of key importance in Bayesian inferential settings , which find applications in domains as diverse as natural language processing ( Xiao & Wang , 2019 ) , stochastic processes ( Rao & Teg , 2013 ) , network analysis ( Perez et al. , 2018 ) or image processing ( Kendall & Gal , 2017 ) , to name only a few . In contrast with model scores , model uncertainties manifest aspects of a system or data generating process that are not exactly known ( Hüllermeier & Waegeman , 2021 ) , and can be decomposed across aleatoric and epistemic components that help scrutinize different aspects in the functioning of a model , and can facilitate interpretability or fairness assessments in important machine learning applications ( Awasthi et al. , 2021 ) . Recently , there has been a growing interest in the study of methods for uncertainty estimation and decomposition ( e.g . Depeweg et al. , 2018 ; Smith & Gal , 2018 ; Van Amersfoort et al. , 2020 ; Tuna et al. , 2021 ) for purposes such as procuring adversarial examples , active learning or out-of-distribution detection . Most importantly , recent work has proposed counterfactual mechanisms for the interpretability of model uncertainties ( Van Looveren & Klaise , 2019 ; Antoran et al. , 2021 ; Schut et al. , 2021 ) , as well as their attribution to individual input features , such as pixels in an image . These methods proceed by identifying small adversarial or in-distribution variations in the raw input , s.t . predictive uncertainties in a model output are reduced . Then , attributions to individual pixels , words or categories are commonly assigned by direct comparison . This can facilitate the understanding of the strengths and weaknesses of varied probabilistic models , however , the optimization task to produce such counterfactuals requires a good balance between reducing uncertainties and minimising changes to original features , which is hard to achieve in practice . Most importantly , these methods do not satisfy commonly desired properties associated with modern importance attribution techniques ( Sundararajan et al. , 2017 ) , such as completeness or implementation invariance . In this paper , we present a novel framework for the attribution of predictive uncertainties , applicable to Bayesian differentiable models . We leverage path integrals ( Sundararajan et al. , 2017 ) along with in-distribution curves ( Jha et al. , 2020 ) , and we propose aggregating attributions over paths starting at a reference counterfactual which bears no predictive uncertainty . We ensure that completeness and additional desirable properties are satisfied , hence , uncertainties are completely explained by ( and decomposed over ) pixels in an image . We validate our approach by direct comparison with recently introduced counterfactual CLUE explanations ( Antoran et al. , 2021 ) , as well as popular interpretability methods , such as integrated gradients ( Sundararajan et al. , 2017 ) , LIME ( Ribeiro et al. , 2016 ) or kernelSHAP ( Lundberg & Lee , 2017 ) , which we adapt for the attribution of predictive uncertainties . Experiments1on benchmark image data sets show that , in comparison to competing alternatives , our proposed method offers sparse and easily interpretable attributions , always limited to relevant super-pixels . Thus , we offer improved means to understand the interplay between raw inputs and aleatoric/epistemic uncertainties in deep models . 2 UNCERTAINTY ATTRIBUTIONS . We focus our presentation on a classification task with a neural classifier f : Rn×W → ∆|C|−1 of a fixed architecture . The classifier maps feature vectors x ∈ Rn along with network weights w ∈ W to an element in the standard ( |C| − 1 ) -simplex , which represents membership probabilities across classes in a set C. On training f within an ( approximate ) Bayesian setting , we commonly obtain a posterior over the hypothesis space of models , i.e . a distribution π ( w|D ) over weights conditioned on the available train data D = { xi , ci } i=1,2 , .... Popular approaches to procure such posterior often differ in their approach to incorporate prior knowledge and include dropout ( Srivastava et al. , 2014 ) , Bayes-by-Backprop ( Blundell et al. , 2015 ) or SG-HMC ( Springenberg et al. , 2016 ) . A model score for classification with a new data point x⋆ ∈ Rn is derived from the posterior predictive distribution by marginalising over posterior weights , i.e . π ( x⋆|D ) = ∫ W f ( x⋆ , w ) π ( w|D ) dw = Ew|D [ f ( x⋆ , w ) ] , ( 1 ) and is easily approximated as 1N ∑N i=1 f ( x ⋆ , wi ) , with weight samples wi ∼ π ( w|D ) , i = 1 , . . . , N . In the following , we are concerned with the entropy as a measure of uncertainty : H ( x|D ) = − ∑ c∈C Ew|D [ fc ( x , w ) ] · logEw|D [ fc ( x , w ) ] ( 2 ) where fc ( x , w ) represents the probability of class-c membership . Remark . Concepts in this paper trivially extend to varied representations of uncertainty in classification and regression settings . Details are omitted for simplicity in the presentation . The entropy term in ( 2 ) may further be decomposed through the law of iterated variances ( Kendall & Gal , 2017 ) so as to yield an aleatoric term Ha ( x|D ) = Ew|D [ H ( x , w ) ] = − ∑ c∈C Ew|D [ fc ( x , w ) · log fc ( x , w ) ] , which measures the mean predictive entropy across models in the posterior hypothesis space , as well as the mutual information or epistemic term , He ( x|D ) = H ( x|D ) −Ha ( x|D ) that represents model uncertainty projected into the latent membership vector π ( x|D ) . Intuitively , aleatoric uncertainty represents natural stochastic variation in the observations over repeated experiments ; on the other hand , epistemic uncertainty is descriptive of model unknowns due to inadequate data or inappropriate modelling choices . 2.1 PATH INTEGRATED GRADIENTS . Path integrated gradients ( IG ) ( Sundararajan et al. , 2017 ) is a simple and popular method for importance attributions that differs from conventional feature removal and permutation techniques ( Covert et al. , 2020 ) , and is primarily targeted at image processing tasks . It is a practical and easy to implement alternative to layer-wise relevance propagation Montavon et al . ( 2019 ) or DeepLift ( Shrikumar et al. , 2017 ) ; it retains desired properties including sensitivity and implementation invariance , and has been extended to several adaptations ( Smilkov et al. , 2017 ; Jha et al. , 2020 ) . Given a classifier f ( · , w ) along with a feature vector x , path IG explains a model score f ( x , w ) using an alternative fiducial vector x0 as a reference , which is presumably not associated with any 1Source code for reproducing our results can be found at the following link : [ removed for blind review ] class observed in the training data . The attributed importance at index or pixel i is given by IGδi ( x|w ) = ∫ 1 0 ∂f ( δ ( α ) , w ) ∂δi ( α ) ∂δi ( α ) ∂α dα so that ∑ i IGi ( x|w ) = f ( x , w ) − f ( x0 , w ) . The result follows from the fundamental theorem of calculus for line integrals known as the gradient theorem . Here , δ : [ 0 , 1 ] → Rn represents a curve with endpoints at δ ( 0 ) = x0 and δ ( 1 ) = x . In vanilla IG , δ is parametrized as a straight path between fiducial and image feature vectors , i.e . δ ( α ) = x0 + α ( x− x0 ) , and the above simplifies to IGi ( x|w ) = ( xi − x0i ) × ∫ 1 0 ∂f ( x0 + α ( x− x0 ) , w ) ∂xi dα , so that importances are heavily influenced by differences in pixel values between x and x0 . However , a straight line often transitions the path x0 ⇝ x out-of-distribution or outside the datamanifold ( Jha et al. , 2020 ; Adebayo et al. , 2020 ) . Also , the fiducial choice is considered problematic ( Sundararajan et al. , 2017 ) and generally defaults to a black background . 2.2 INTEGRATED GRADIENTS WITH UNCERTAINTY . Commonly , the classifier f ( · , w ) is presumed to be binary ( Sundararajan et al. , 2017 ) with model scores constrained to the interval [ 0 , 1 ] . However , the above logic for importance attribution does easily generalize to multi-class Bayesian settings in the presence of uncertainty . Here , the posterior predictive classifier π ( x|D ) introduced in ( 1 ) accepts a path IG importance at index i given by IGδi ( x ) = ∫ 1 0 Ew|D [ ∂f ( δ ( α ) , w ) ∂δi ( α ) ] ∂δi ( α ) ∂α dα , which represents a mean-average trajectory over the curve δ and follows from dominated convergence . Most importantly , we may employ path IG to explain univariate measures of uncertainty from the posterior predictive distribution over classes , as observed in Figure 1 . Here , IG-Hδi ( x ) = − ∑ c∈C ∫ 1 0 ∆i ( α ) ∂δi ( α ) ∂α dα ( 3 ) captures variations in predictions covering multiple classes , and is defined s.t . ∆i ( α ) = ( 1 + logEw|D [ fc ( δ ( α ) , w ) ] ) · Ew|D [ ∂fc ( δ ( α ) , w ) ∂δi ( α ) ] attributes importances for the change in entropy between a fiducial point and a feature vector , and ∆i ( α ) = Ew|D [ ( 1 + log fc ( δ ( α ) , w ) ) · ∂fc ( δ ( α ) , w ) ∂δi ( α ) ] is the analogue representation restricted to the aleatoric term . Any variation in epistemic uncertainty is readily shown to be explained as the difference in importances between the above two terms . In Figure 1 , the goal is not to understand why the classifier suggests this picture refers to a dog ; instead , we comprehend why the model struggles to predict any single class with confidence , and we notice that the leash and a human hand are problematic . Further examples may be found in Appendix A ; in all cases , importances have been smoothed with a Gaussian filter , averaging over positive ( increasing uncertainty ) and negative ( decreasing uncertainty ) contributions . The attributions are easily computed by standard Bayesian procedures , approximating the inner expectations with simulations , however , the choice of fiducial ( black screen ) and out-of-distribution path remain controversial and a significant challenge in order to enable an intuitive understanding of predictive uncertainties in our model , which may represent a barrier in applications ( Antoran et al. , 2021 ) . 3 METHODOLOGY . Next , we describe the computational process summarized in Algorithm 1 , which produces novel in-distribution attributions of uncertainty for a feature vector x and predictive posterior π ( x|D ) in ( 1 ) . We do so through the use of a counterfactual fiducial bearing no relation to causal inference ( Pearl , 2010 ) . This counterfactual is an alternative vector x0 defined similarly to CLUEs in Antoran et al . ( 2021 ) , i.e . ( i ) in distribution and ( ii ) close to x according to some arbitrary distance metric . However , we furthermore require that the class distribution π ( x0|D ) bears close to 0 predictive uncertainty . Intuitively , we construct IG attributions using finely tuned fiducial points , by comparing ambiguous images to easily predicted counterparts that bear a significant resemblance . Algorithm 1 : Uncertainty attributions input : Feature vector x , predictive posterior π ( ·|D ) and uncertainty estimator H ( · ) . Distance metric d ( · , · ) , VAE encoder ϕ ( · ) and decoder ψ ( · ) . Penalty λ > > 0 and learning rate ν > 0. output : Uncertainty Attributions IG-Hi ( x ) , i = 1 , . . . , n. Initialise z0 = z = ϕµ ( x ) ; Compute predicted class ĉ = argmaxi πi ( x|D ) ; while L not converged do L ← d ( ψ ( z0 ) , x ) + 12m ∑ j z 2 j + λ log πĉ ( ψ ( z ) |D ) and z0 ← z0 − ν∇zL ; end while L not converged do L ← d ( ψ ( z ) , x ) + 12m ∑ j z 2 j and z ← z − ν∇zL ; end Approximate IG-Hδi ( x ) , i = 1 , . . . , n in ( 5 ) along δz0→z through trapezoidal integration . To begin with , we assume the existence of a generative variational auto-encoder ( VAE ) composed of an encoder ϕ : Rn → Rm and decoder ψ : Rm → Rn . As customary , the data-generating process in Rm is unit-Gaussian with an arbitrary dimensionality m < < n .

Summary: The authors propose a method to interpret model uncertainty for individual input examples. That is, it attempts to identify which input features (e.g. pixels of an image) are contributing to prediction uncertainty (measured as entropy in the multi-class prediction probabilities). To do this, for some query input, the proposed method applies integrated gradients between the input and a "fiducial" (i.e. high certainty) reference, which serves as a version of the input which is similar, but the network is very certain of its prediction class (the same prediction class as the query input). To find this fiducial reference, the method uses a pretrained variational autoencoder to find an input in the latent embedding space of high prediction certainty, that also lies closely on the latent embedding manifold. Integrated gradient interpolation is done linearly in this latent space. The authors show through several examples that the explanations for uncertainty are reasonable, and compare them to other methods like CLUE, LIME, and SHAP.