Path Integrals for the Attribution of Model Uncertainties

1 INTRODUCTION . Estimating and understanding model uncertainties is of key importance in Bayesian inferential settings , which find applications in domains as diverse as natural language processing ( Xiao & Wang , 2019 ) , stochastic processes ( Rao & Teg , 2013 ) , network analysis ( Perez et al. , 2018 ) or image processing ( Kendall & Gal , 2017 ) , to name only a few . In contrast with model scores , model uncertainties manifest aspects of a system or data generating process that are not exactly known ( Hüllermeier & Waegeman , 2021 ) , and can be decomposed across aleatoric and epistemic components that help scrutinize different aspects in the functioning of a model , and can facilitate interpretability or fairness assessments in important machine learning applications ( Awasthi et al. , 2021 ) . Recently , there has been a growing interest in the study of methods for uncertainty estimation and decomposition ( e.g . Depeweg et al. , 2018 ; Smith & Gal , 2018 ; Van Amersfoort et al. , 2020 ; Tuna et al. , 2021 ) for purposes such as procuring adversarial examples , active learning or out-of-distribution detection . Most importantly , recent work has proposed counterfactual mechanisms for the interpretability of model uncertainties ( Van Looveren & Klaise , 2019 ; Antoran et al. , 2021 ; Schut et al. , 2021 ) , as well as their attribution to individual input features , such as pixels in an image . These methods proceed by identifying small adversarial or in-distribution variations in the raw input , s.t . predictive uncertainties in a model output are reduced . Then , attributions to individual pixels , words or categories are commonly assigned by direct comparison . This can facilitate the understanding of the strengths and weaknesses of varied probabilistic models , however , the optimization task to produce such counterfactuals requires a good balance between reducing uncertainties and minimising changes to original features , which is hard to achieve in practice . Most importantly , these methods do not satisfy commonly desired properties associated with modern importance attribution techniques ( Sundararajan et al. , 2017 ) , such as completeness or implementation invariance . In this paper , we present a novel framework for the attribution of predictive uncertainties , applicable to Bayesian differentiable models . We leverage path integrals ( Sundararajan et al. , 2017 ) along with in-distribution curves ( Jha et al. , 2020 ) , and we propose aggregating attributions over paths starting at a reference counterfactual which bears no predictive uncertainty . We ensure that completeness and additional desirable properties are satisfied , hence , uncertainties are completely explained by ( and decomposed over ) pixels in an image . We validate our approach by direct comparison with recently introduced counterfactual CLUE explanations ( Antoran et al. , 2021 ) , as well as popular interpretability methods , such as integrated gradients ( Sundararajan et al. , 2017 ) , LIME ( Ribeiro et al. , 2016 ) or kernelSHAP ( Lundberg & Lee , 2017 ) , which we adapt for the attribution of predictive uncertainties . Experiments1on benchmark image data sets show that , in comparison to competing alternatives , our proposed method offers sparse and easily interpretable attributions , always limited to relevant super-pixels . Thus , we offer improved means to understand the interplay between raw inputs and aleatoric/epistemic uncertainties in deep models . 2 UNCERTAINTY ATTRIBUTIONS . We focus our presentation on a classification task with a neural classifier f : Rn×W → ∆|C|−1 of a fixed architecture . The classifier maps feature vectors x ∈ Rn along with network weights w ∈ W to an element in the standard ( |C| − 1 ) -simplex , which represents membership probabilities across classes in a set C. On training f within an ( approximate ) Bayesian setting , we commonly obtain a posterior over the hypothesis space of models , i.e . a distribution π ( w|D ) over weights conditioned on the available train data D = { xi , ci } i=1,2 , .... Popular approaches to procure such posterior often differ in their approach to incorporate prior knowledge and include dropout ( Srivastava et al. , 2014 ) , Bayes-by-Backprop ( Blundell et al. , 2015 ) or SG-HMC ( Springenberg et al. , 2016 ) . A model score for classification with a new data point x⋆ ∈ Rn is derived from the posterior predictive distribution by marginalising over posterior weights , i.e . π ( x⋆|D ) = ∫ W f ( x⋆ , w ) π ( w|D ) dw = Ew|D [ f ( x⋆ , w ) ] , ( 1 ) and is easily approximated as 1N ∑N i=1 f ( x ⋆ , wi ) , with weight samples wi ∼ π ( w|D ) , i = 1 , . . . , N . In the following , we are concerned with the entropy as a measure of uncertainty : H ( x|D ) = − ∑ c∈C Ew|D [ fc ( x , w ) ] · logEw|D [ fc ( x , w ) ] ( 2 ) where fc ( x , w ) represents the probability of class-c membership . Remark . Concepts in this paper trivially extend to varied representations of uncertainty in classification and regression settings . Details are omitted for simplicity in the presentation . The entropy term in ( 2 ) may further be decomposed through the law of iterated variances ( Kendall & Gal , 2017 ) so as to yield an aleatoric term Ha ( x|D ) = Ew|D [ H ( x , w ) ] = − ∑ c∈C Ew|D [ fc ( x , w ) · log fc ( x , w ) ] , which measures the mean predictive entropy across models in the posterior hypothesis space , as well as the mutual information or epistemic term , He ( x|D ) = H ( x|D ) −Ha ( x|D ) that represents model uncertainty projected into the latent membership vector π ( x|D ) . Intuitively , aleatoric uncertainty represents natural stochastic variation in the observations over repeated experiments ; on the other hand , epistemic uncertainty is descriptive of model unknowns due to inadequate data or inappropriate modelling choices . 2.1 PATH INTEGRATED GRADIENTS . Path integrated gradients ( IG ) ( Sundararajan et al. , 2017 ) is a simple and popular method for importance attributions that differs from conventional feature removal and permutation techniques ( Covert et al. , 2020 ) , and is primarily targeted at image processing tasks . It is a practical and easy to implement alternative to layer-wise relevance propagation Montavon et al . ( 2019 ) or DeepLift ( Shrikumar et al. , 2017 ) ; it retains desired properties including sensitivity and implementation invariance , and has been extended to several adaptations ( Smilkov et al. , 2017 ; Jha et al. , 2020 ) . Given a classifier f ( · , w ) along with a feature vector x , path IG explains a model score f ( x , w ) using an alternative fiducial vector x0 as a reference , which is presumably not associated with any 1Source code for reproducing our results can be found at the following link : [ removed for blind review ] class observed in the training data . The attributed importance at index or pixel i is given by IGδi ( x|w ) = ∫ 1 0 ∂f ( δ ( α ) , w ) ∂δi ( α ) ∂δi ( α ) ∂α dα so that ∑ i IGi ( x|w ) = f ( x , w ) − f ( x0 , w ) . The result follows from the fundamental theorem of calculus for line integrals known as the gradient theorem . Here , δ : [ 0 , 1 ] → Rn represents a curve with endpoints at δ ( 0 ) = x0 and δ ( 1 ) = x . In vanilla IG , δ is parametrized as a straight path between fiducial and image feature vectors , i.e . δ ( α ) = x0 + α ( x− x0 ) , and the above simplifies to IGi ( x|w ) = ( xi − x0i ) × ∫ 1 0 ∂f ( x0 + α ( x− x0 ) , w ) ∂xi dα , so that importances are heavily influenced by differences in pixel values between x and x0 . However , a straight line often transitions the path x0 ⇝ x out-of-distribution or outside the datamanifold ( Jha et al. , 2020 ; Adebayo et al. , 2020 ) . Also , the fiducial choice is considered problematic ( Sundararajan et al. , 2017 ) and generally defaults to a black background . 2.2 INTEGRATED GRADIENTS WITH UNCERTAINTY . Commonly , the classifier f ( · , w ) is presumed to be binary ( Sundararajan et al. , 2017 ) with model scores constrained to the interval [ 0 , 1 ] . However , the above logic for importance attribution does easily generalize to multi-class Bayesian settings in the presence of uncertainty . Here , the posterior predictive classifier π ( x|D ) introduced in ( 1 ) accepts a path IG importance at index i given by IGδi ( x ) = ∫ 1 0 Ew|D [ ∂f ( δ ( α ) , w ) ∂δi ( α ) ] ∂δi ( α ) ∂α dα , which represents a mean-average trajectory over the curve δ and follows from dominated convergence . Most importantly , we may employ path IG to explain univariate measures of uncertainty from the posterior predictive distribution over classes , as observed in Figure 1 . Here , IG-Hδi ( x ) = − ∑ c∈C ∫ 1 0 ∆i ( α ) ∂δi ( α ) ∂α dα ( 3 ) captures variations in predictions covering multiple classes , and is defined s.t . ∆i ( α ) = ( 1 + logEw|D [ fc ( δ ( α ) , w ) ] ) · Ew|D [ ∂fc ( δ ( α ) , w ) ∂δi ( α ) ] attributes importances for the change in entropy between a fiducial point and a feature vector , and ∆i ( α ) = Ew|D [ ( 1 + log fc ( δ ( α ) , w ) ) · ∂fc ( δ ( α ) , w ) ∂δi ( α ) ] is the analogue representation restricted to the aleatoric term . Any variation in epistemic uncertainty is readily shown to be explained as the difference in importances between the above two terms . In Figure 1 , the goal is not to understand why the classifier suggests this picture refers to a dog ; instead , we comprehend why the model struggles to predict any single class with confidence , and we notice that the leash and a human hand are problematic . Further examples may be found in Appendix A ; in all cases , importances have been smoothed with a Gaussian filter , averaging over positive ( increasing uncertainty ) and negative ( decreasing uncertainty ) contributions . The attributions are easily computed by standard Bayesian procedures , approximating the inner expectations with simulations , however , the choice of fiducial ( black screen ) and out-of-distribution path remain controversial and a significant challenge in order to enable an intuitive understanding of predictive uncertainties in our model , which may represent a barrier in applications ( Antoran et al. , 2021 ) . 3 METHODOLOGY . Next , we describe the computational process summarized in Algorithm 1 , which produces novel in-distribution attributions of uncertainty for a feature vector x and predictive posterior π ( x|D ) in ( 1 ) . We do so through the use of a counterfactual fiducial bearing no relation to causal inference ( Pearl , 2010 ) . This counterfactual is an alternative vector x0 defined similarly to CLUEs in Antoran et al . ( 2021 ) , i.e . ( i ) in distribution and ( ii ) close to x according to some arbitrary distance metric . However , we furthermore require that the class distribution π ( x0|D ) bears close to 0 predictive uncertainty . Intuitively , we construct IG attributions using finely tuned fiducial points , by comparing ambiguous images to easily predicted counterparts that bear a significant resemblance . Algorithm 1 : Uncertainty attributions input : Feature vector x , predictive posterior π ( ·|D ) and uncertainty estimator H ( · ) . Distance metric d ( · , · ) , VAE encoder ϕ ( · ) and decoder ψ ( · ) . Penalty λ > > 0 and learning rate ν > 0. output : Uncertainty Attributions IG-Hi ( x ) , i = 1 , . . . , n. Initialise z0 = z = ϕµ ( x ) ; Compute predicted class ĉ = argmaxi πi ( x|D ) ; while L not converged do L ← d ( ψ ( z0 ) , x ) + 12m ∑ j z 2 j + λ log πĉ ( ψ ( z ) |D ) and z0 ← z0 − ν∇zL ; end while L not converged do L ← d ( ψ ( z ) , x ) + 12m ∑ j z 2 j and z ← z − ν∇zL ; end Approximate IG-Hδi ( x ) , i = 1 , . . . , n in ( 5 ) along δz0→z through trapezoidal integration . To begin with , we assume the existence of a generative variational auto-encoder ( VAE ) composed of an encoder ϕ : Rn → Rm and decoder ψ : Rm → Rn . As customary , the data-generating process in Rm is unit-Gaussian with an arbitrary dimensionality m < < n .

Summary: In this paper, the authors consider the question of how to explain model uncertainty in terms of the input features. Whereas prior work (e.g., CLUE) uses a counterfactual approach, the authors develop an attribution-based approach (assigning a score to each feature). The particular method is built on Integrated Gradients (IG): a baseline is identified that resembles the input but has no prediction uncertainty, and a non-straightline path is followed by traversing a VAE's latent space.  The experiments suggest that the proposed method (IG-H) has some advantages relative to CLUE, as well as simple adaptations of IG/LIME/SHAP to uncertainty attribution.