A Learning Theoretic Perspective on Local Explainability

1 INTRODUCTION . There has been a growing interest in interpretable machine learning , which seeks to help people better understand their models . While interpretable machine learning encompasses a wide range of problems , it is a fairly uncontroversial hypothesis that there exists a trade-off between a model ’ s complexity and general notions of interpretability . This hypothesis suggests a seemingly natural connection to the field of learning theory , which has thoroughly explored relationships between a function class ’ s complexity and generalization . However , formal connections between interpretability and learning theory remain relatively unstudied . Though there are several notions of conveying interpretability , one common and flexible approach is to use local approximations . Formally , local approximation explanations ( which we will refer to as “ local explanations ” ) provide insight into a model ’ s behavior as follows : for any black-box model f ∈ F and input x , the explanation system produces a simple function , gx ( x′ ) ∈ Glocal , which approximates f in a chosen neighborhood , x′ ∼ Nx . Crucially , the freedom to specify both Glocal andNx grants local explanations great versatility . In this paper , we provide two connections between learning theory and how well f can be approximated locally ( i.e . the fidelity of local explanations ) . Our first result studies the standard problem of performance generalization by relating test-time predictive accuracy to a notion of local explainability . As it turns out , our focus on local explanations leads us to unique tools and insights from a learning theory point of view . Our second result identifies and addresses an unstudied – yet important – question regarding explanation generalization . This question pertains to a growing class of explanation systems , such as MAPLE ( Plumb et al. , 2018 ) and RL-LIM ( Yoon et al. , 2019 ) , which we call finite sample-based local explanations1 . These methods learn their local approximations using a common finite sample drawn from data distribution D ( in contrast to canonical local approximation methods such as LIME ( Ribeiro et al. , 2016 ) ) and , as a result , run the risk of overfitting to this finite sample . In light of this , we answer the following question : for these explanation-learning systems , how well do the quality of local explanations generalize to data not seen during training ? ∗Denotes equal contribution 1This terminology is not to be confused with “ example-based explanations ” where the explanation itself is in the form of data instances rather than a function . We address these questions with two bounds , which we outline now . Regarding performance generalization , we derive our first main result , Theorem 1 , which bounds the expected test mean squared error ( MSE ) of any f in terms of its MSE over the m samples in the training set , S = { ( xi , yi ) } mi=1 : E ( x , y ) ∼D [ ( f ( x ) − y ) 2 ] ︸ ︷︷ ︸ Test MSE ≤ Õ ( 1 m m∑ i=1 ( f ( xi ) − yi ) 2︸ ︷︷ ︸ Train MSE +E x∼D , x′∼Nx [ ( gx′ ( x ) − f ( x ) ) 2 ] ︸ ︷︷ ︸ Interpretability Term ( MNF ) + ρSR̂S ( Glocal ) ︸ ︷︷ ︸ Complexity Term ) Regarding explanation generalization for finite sample-based explanation-learning systems , we apply a similar proof technique to obtain Theorem 2 , which bounds the quality of the system ’ s explanations on unseen data in terms of their quality on the data on which the system was trained : E x∼D , x′∼Nx [ ( gx′ ( x ) − f ( x ) ) 2 ] ︸ ︷︷ ︸ Test MNF ≤ 1 m m∑ i=1 Ex′∼Nxi [ ( f ( xi ) − gx′ ( xi ) ) 2 ] ︸ ︷︷ ︸ Train MNF + Õ ( ρSR̂S ( Glocal ) ) ︸ ︷︷ ︸ Complexity Term Before summarizing our contributions , we discuss the key terms and their relationships . • Interpretability terms : The terms involving MNF correspond to Mirrored Neighborhood Fidelity , a metric we use to measure local explanation quality . As we discuss in Section 3 , this is a reasonable modification of the commonly used Neighborhood Fidelity ( NF ) metric ( Ribeiro et al. , 2016 ; Plumb et al. , 2018 ) . Intuitively , we generally expect MNF to be larger when the neighborhood sizes are larger since each gx′ is required to extrapolate farther . • Complexity term : This term measures the complexity of the local explanation system g in terms of ( a ) the complexity of the local explanation class Glocal and ( b ) ρS , a quantity that we define and refer to as the neighborhood disjointedness factor . As we discuss in Section 4 , ρS is a value in [ 1 , √ m ] ( where m = |S| ) that is proportional to the level of disjointedness of the neighborhoods for points in the sample S. Intuitively , we expect ρS to be larger when the neighborhoods sizes are smaller since smaller neighborhoods will overlap less . Notably , both our bounds capture the following key trade-off : as neighborhood widths increase , MNF increases but ρS decreases . As such , our bounds are non-trivial only if the neighborhoods Nx can be chosen such that MNF remains small but ρS grows slower than Õ ( √ m ) ( since R̂S ( Glocal ) typically decays as Õ ( 1/ √ m ) ) . We summarize our main contributions as follows : ( 1 ) We make a novel connection between performance generalization and local explainability , arriving at Theorem 1 . Given the relationship between MNF and ρS , this bound roughly captures that an easier-to-interpret f enjoys better generalization guarantees , a potentially valuable result when reasoning about F is difficult ( e.g . for neural networks ) . Further , our proof technique may be of independent theoretical interest as it provides a new way to bound the Rademacher complexity of a particular class of randomized functions ( see Section 4 ) . ( 2 ) We motivate and explore an important generalization question about expected explanation quality . Specifically , we arrive at Theorem 2 , a bound for test MNF in terms of train MNF . This bound suggests that practitioners can better guarantee good local explanation quality ( measured by MNF ) using methods which encourage the neighborhood widths to be wider ( see Section 5 ) . ( 3 ) We verify empirically on UCI Regression datasets that our results non-trivially reflect the two types of generalization in practice . First , we demonstrate that ρS can indeed exhibit slower than Õ ( √ m ) growth without significantly increasing the MNF terms . Also , for Theorem 2 , we show that the generalization gap indeed improves with larger neighborhoods ( see Section 6 ) . ( 4 ) Primarily to aid in our theoretical results , we propose MNF as a novel yet reasonable measure of local explainability . Additionally , we argue that this metric presents a promising avenue for future study , as it may naturally complement NF and offer a unique advantage when evaluating local explanations on “ realistic ” on-distribution data ( see Section 3 ) . 2 RELATED WORK . Interpretability meets learning theory . Semenova et al . ( 2020 ) study the performance generalization of models learned from complex classes when they can be globally well-approximated by simpler ( e.g . interpretable ) classes . In such cases , their theory argues that if the complex class has many models that perform about as optimally on training data , generalization from the complex class can be more closely bounded using the simpler class ’ s complexity . In our corresponding results , we similarly aim to avoid involving the larger class ’ s complexity . However , we directly study generalization via a model ’ s local explainability , rather than instantiate ” complex ” and “ simple ” classes for global approximations . The two are fundamentally different technical problems ; standard learning theory results can not be directly applied as they are done for single-function global approximations . Statistical localized regression . ( Fan , 1993 ; Fan & Gijbels , 1996 ) are canonical results which bound the squared error of a nonparametric function defined using locally fit models . These local models are both simple ( e.g . linear ) and similarly trained by weighting real examples with a kernel ( i.e . an implied neighborhood ) . However , each local model is only used to make a prediction at its source point and the theory requires shrinking the kernel width towards 0 as the sample size grows . We instead fit local models as explanations for a trained model ( i.e . which is considered the “ true regression function ” ) and more importantly , care about the fidelity of each local model over whole ( non-zero ) neighborhoods . Unlike localized regression , this allows us to use uniform convergence to bound test error with empirical and generalization terms . While the previous results do not have empirical terms , the learning rates are exponential in the number of samples . Learning Theory . Another line of related work also studies how to explain generalization of overparameterized classes . As standard uniform convergence on these classes often leads to vacuous bounds , a general approach that has followed from ( Nagarajan & Kolter , 2019 ; Zhang et al. , 2017 ; Neyshabur et al. , 2014 ) has been to study implications of different biases placed on learned models . We study what would happen if an overparameterized model had an unexplored type of bias , one that is inspired by local explainability . Additionally , our work ’ s technical approach parallels another line of existing results which likewise try to apply uniform convergence on separate surrogate classes . This includes PAC-Bayesian bounds , a large family of techniques that come from looking at a stochastic version of a model in parameter space ( McAllester , 1998 ; 2003 ; Langford & Caruana , 2002 ; Langford & Shawe-Taylor , 2003 ) . In a different vein , some results in deep learning look at compressed , sparsified , or explicitly regularized surrogates of neural networks ( Arora et al. , 2018 ; Dziugaite & Roy , 2017 ) . In our case , the surrogate class is a collection of local explanations . 3 MIRRORED NEIGHBORHOOD FIDELITY . In order to connect local explanations to generalization , recall that we study a measure of local interpretability which we call “ mirrored neighborhood fidelity ” ( MNF ) . As we explain below , this quantity comes from a slight modification to an existing metric for local approximation explanations , namely , that of neighborhood fidelity ( NF ) . To define these quantities , we use the following notation . Let X be an input space and D be a distribution over X × Y where Y ⊆ R. Then , let F be a class of functions f : X → Y . For our theoretical results , we specifically assume that Y is bounded as Y = [ −B , B ] for some B > 0 ( though this does not matter for the remainder of this section ) . In order to provide local explanations , we need to fix a nearby region around each x ∈ X . To this end , for any x , letNx correspond to some distribution denoting a local neighborhood at x ( e.g . typically chosen to be a distribution centered at x ) . For any distribution N , we use pN ( x ) to denote its density at x . Finally , let G be a class of local explainers g : X × X → Y such that for each x ∈ X , the local explanation g ( x , · ) : X → Y belongs to a class of ( simple ) functions ( e.g . linear ) , Glocal . For convenience , we denote g ( x , · ) as gx ( · ) and use it to locally approximate f in the neighborhood defined by Nx . The accuracy of the local explanation system g is usually quantified by a term called “ neighbhorhood fidelity ” which is defined as follows ( Ribeiro et al. , 2016 ; 2018 ; Plumb et al. , 2018 ; 2020 ) : NF ( f , g ) : = Ex∼D [ Ex′∼Nx [ ( f ( x′ ) − gx ( x′ ) ) 2 ] ] . To verbally interpet this , let us call x as the “ source ” point which gives rise to a local explanation gx ( · ) and x′ the “ target ” points that we try to fit using g. To compute NF ( f , g ) , we need to do the following : for each source point x , we first compute the average error in the fit of gx ( · ) over draws of nearby target points x′ ∼ Nx ; then , to get a more overall measure of g ’ s quality , we globally average this error across draws of the source point x ∼ D. Now , to define MNF , we take the same expression as for NF but swap x and x′ within the innermost expectation ( without modifying how each variable is distributed ) . In other words , we now sample a target point x from D and sample source points x′ from a distribution over points near a given x . Since this local distribution is over source points rather than target points , just for the sake of distinguishing , we ’ ll refer to this as a mirrored neighborhood distribution and denote it as Nmirx . We define MNF more formally below , following which we explain how to better understand it : Definition 3.1 . ( Mirrored Neighborhood Fidelity ) We define MNF : F × G → R as MNF ( f , g ) : = Ex∼D [ Ex′∼Nmirx [ ( f ( x ) − gx′ ( x ) ) 2 ] ] . and with an abuse of notation , we let MNF ( f , g , x ) : = Ex′∼Nmirx [ ( f ( x ) − gx′ ( x ) ) 2 ] . Understanding MNF . It is helpful to parse the expression for MNF in two different ways . First , we can think of it as measuring the error in approximating every target point x ∈ X through a randomized locally-approximating function gx′ ( · ) , where x′ is randomly drawn from the neighborhood Nmirx . A second way to parse this is in a manner similar to how we parsed NF . To do this , first we note that the expectations in MNF can be swapped around and rewritten as : MNF ( f , g ) = Ex′∼D† [ Ex∼N† x′ [ ( f ( x ) − gx′ ( x ) ) 2 ] ] , where D† and N†x′ are suitably defined distributions ( derived in Appendix A ) that can be thought of as modified counterparts of D and Nmirx′ respectively . With this rewritten expression , one can read MNF like NF : for each source point ( now x′ ) , we compute the average error in the fit of the corresponding local function ( gx′ ( · ) ) over target points ( x ) drawn from the source ’ s local neighborhood ( N†x′ ) ; this error is then globally averaged over different draws of the source point ( x ′ ∼ D† ) . Why MNF ? While both NF and MNF are closely related measures of local explanation quality on f , studying MNF allows us to make connections between local explainability and different notions of generalization ( Sections 4 and 5 ) . At a high-level , our results don ’ t apply to NF because the overall distribution of the points that are “ fit ” to calculate NF ( i.e. , the target points ) is not the same as the test data distribution D , instead being D perturbed by Nx . Rather , we prefer this target distribution to be D , which is the case for MNF , for us to be able to neatly bound test-time quantities via a local explainability term . Otherwise , we would have to end up introducing many cumbersome terms . Furthermore , MNF may also be of practical interest to the interpretability community , as it potentially offers a unique advantage over NF when the intended usage of local explanations is centered around understanding how f works on the specific learning task it was trained on . Specifically , we present as an exploratory argument that selecting the target point distribution to be D rather than D perturbed byNx ( as for NF ) may better emphasize the ability of g to approximate f at realistic input points . This is relevant for ML ( and deep learning especially ) because ( a ) high-dimensional datasets often exhibit significant feature dependencies and adherence to lower dimensional manifolds ; ( b ) f can often be highly unpredictable and unstable when extrapolating beyond training data . Thus , when one measures NF with standard neighborhood choices that ignore feature dependencies ( e.g . most commonly Nx = N ( x , σI ) ) , the resulting target distribution may concentrate significantly on regions that are non-relevant to the original learning task at hand . As we illustrate in a toy example , this can lead to overemphasis on fitting noisy off-manifold behavior , deteriorating the fit of explanations relative to task-relevant input regions ( we defer a more detailed discussion of this point and of other trade-offs between NF and MNF to Appendix A ) . 4 GENERALIZATION OF MODEL PERFORMANCE VIA MNF The generalization error of the function f is typically bounded by some notion of the representational complexity of F . While standard results bound complexity in terms of parameter count , there is theoretical value in deriving bounds involving other novel terms . By doing so , one might understand how regularizing for those terms can affect the representation capacity , and in turn , the generalization error of f . Especially when f ’ s complexity may be intractable to bound on its own , introducing these terms provides a potentially useful new way to understand f ’ s generalization . Here specifically , we are interested in establishing a general connection between the representation complexity and the local explainability of any f . This naturally requires coming up with a notion that appropriately quantifies the complexity of G , which we discuss in the first part of this section . As we shall see , G ’ s complexity can be expressed in terms of Glocal , which is generally less complex and more amenable to standard analysis than F in practical settings where interpretability is desired . In the second part , we then relate this quantity to the generalization of f to derive our first main result . Key technical challenge : bounding the complexity of G. The overall idea behind how one could tie the notions of generalization and local explainability is fairly intuitive . For example , consider a simplified setting where we approximate f by dividingX intoK disjoint pieces , i.e . neighborhoods , and then approximating each neighborhood via a simple ( say , linear ) model . Then , one could bound the generalization error of f as the sum of two quantities : first , the error in approximating f via the piecewise linear model , and second , a term involving the complexity of said piecewise model . It is straightforward to show that this complexity term grows polynomially with the piece-count , K , and also the complexity of the simple local approximator ( see Appendix C.0.1 ) . Similarly , one could hope to bound the generalization error of f in terms of MNF ( f , g ) and the complexity of G. However , the key challenge here is that the class G is a much more complex class than the above class of piecewise linear models . For example , a straightforward piece-count-based complexity bound would be infinitely large since there are effectively infinitely many unique pieces in g. Our core technical contribution here is to bound the ( Rademacher ) complexity of G in this more flexible local explanation setting . At a high level , the resulting bound ( to be stated shortly ) grows inversely with “ the level of overlap ” between the neighborhoods { Nmirx |x ∈ X } , quantified as : Definition 4.1 . ( Neighborhood disjointedness factor ) Given a dataset S ∈ ( X × Y ) m , we define the neighborhood disjointedness factor ρS as ρS : = ∫ x′∈X √√√√ 1 m m∑ i=1 ( pNmirxi ( x′ ) ) 2dx′ Understanding the disjointedness factor . ρS can be interpreted as bounding the “ effective number ” of pieces induced by the set of neighborhood distributions { Nmirx |x ∈ X } . This turns out to be a quantity that lies in [ 1 , √ m ] ( shown formally in Appendix Fact B.1 ) . To reason more intuitively about this quantity , it is helpful to consider its behavior in extreme scenarios . First , consider the case where Nmirx is the same distribution ( say N ) regardless of x ; i.e. , neighborhoods are completely overlapping . Then , ρS = ∫ x′∈X ( pN ( x ′ ) ) dx′ = 1 . In the other extreme , consider if neighborhoods centered on the training data are all disjoint with supports X1 , . . . , X|S| . Here , the integral splits into m summands as : ρS = ∑m i=1 ∫ x′∈Xi 1√ m pNmirxi ( x′ ) dx′ = √ m. Thus , ρS grows from 1 to √ m as the level of overlap between the neighborhoods Nmirx1 , . . . , N mir x|S| reduces . For intuition at non-extreme values , we show in Appendix B.2 that in a simple setting , ρ = √ m1−k ( where 0 ≤ k ≤ 1 ) if every neighborhood is just large enough to encompass a 1/m1−k fraction of the mass of D. Rademacher complexity of G. We now use ρS to bound the empirical Rademacher complexity of G. Recall that the empirical Rademacher complexity of a function classH consisting of h : X → R is defined as R̂S ( H ) : = E~σ [ 1 m suph∈H σih ( xi ) ] , where the σi ’ s are i.i.d . and drawn uniformly from { −1 , 1 } . Roughly , this captures the complexity ofH by measuring how well it can fit random labels . Standard results allow us to then use R̂S ( H ) to bound the generalization error for h ∈ H. Now , in order to define the Rademacher complexity of G ( which consists of a different kind of functions whose domain is X ×X instead of X ) , it is useful to think of g as a randomized function . Specifically , at any target point x , the output of g is a random variable gx′ ( x ) where the randomness comes from x′ ∼ Nmirx . Then , in Lemma 4.1 , we take this stochasticity into account to define and bound the complexity of G. To keep our statement general , we consider a generic loss function L : R×R→ R ( e.g. , the squared error loss is L ( y , y′ ) = ( y − y′ ) 2 ) . Indeed , whenever L satisfies a standard Lipschitz assumption , we can bound the complexity of G ( composed with the loss function L ) in terms of ρS , the complexity of Glocal , and the Lipschitzness of L : Lemma 4.1 . ( see Appendix Lemma D.1 for full , precise statement ) Let L ( · , y′ ) be a c-Lipschitz function w.r.t . y′ in that for all y1 , y2 ∈ [ −B , B ] , |L ( y1 , y′ ) − L ( y2 , y′ ) | ≤ c|y1 − y2| . Then , the empirical Rademacher complexity of G under the loss function L is defined and bounded as : R̂S ( L ◦ G ) : = E~σ [ sup g∈G 1 m m∑ i σiEx′∼Nmirxi [ L ( gx′ ( xi ) , yi ) ] ] ≤ O ( cρSR̂S ( Glocal ) · lnm ) where ~σ is uniformly distributed over { −1 , 1 } m. We note that the proof technique employed here may be of independent theoretical interest as it provides a novel way to bound the complexity of a ( particular type of ) randomized function . Although techniques like PAC-Bayes provide ways to bound the complexity of randomized functions , they only apply to functions where the randomization comes from stochasticity in the parameters , which is not the case here . Main result . With the above key lemma in hand , we are now ready to prove our main result , which bounds the generalization error of f in terms of the complexity of G , thereby establishing a connection between model generalization and local explainability . Theorem 1 . ( see Appendix Theorem 3 for full , precise statement ) With probability over 1 − δ over the draws of S = { ( x1 , y1 ) , . . . , ( xm , ym ) } ∼ Dm , for all f ∈ F and for all g ∈ G , we have ( ignoring ln 1/δ factors ) : E ( x , y ) ∼D [ ( f ( x ) − y ) 2 ] ≤ 4 m m∑ i=1 ( f ( xi ) − yi ) 2 + 2Ex∼D [ Ex′∼Nmirx [ ( f ( x ) − gx′ ( x ) ) 2 ] ] ︸ ︷︷ ︸ MNF ( f , g ) + 4 m m∑ i=1 Ex′∼Nmirx [ ( f ( xi ) − gx′ ( xi ) ) 2 ] ︸ ︷︷ ︸ MNF ( f , g , xi ) +O ( BρSR̂S ( Glocal ) lnm ) . This result decomposes the test error of f into four quantities . The first quantity corresponds to the training error of f on the training set S. The second and the third correspond to the MNF of f with respect to g , computed on test and training data respectively . The fourth and final quantity corresponds to a term that bounds the complexity of G in terms of the “ disjointedness factor ” ρS and the complexity of the local function class Glocal . Takeaway . A key aspect of this bound is the trade-off that it captures when varying neighborhood widths . Consider shrinking the neighborhood widths to smaller and smaller values , in turn creating less and less overlap between the neighborhoods of the training examples in S. Then , on the one hand , we ’ d observe that the complexity term ( the fourth term on the R.H.S ) increases . Specifically , since R̂S ( Glocal ) typically scales as O ( 1/ √ m ) , as we go from the one extreme of full overlap to the other extreme of complete disjointedness , the complexity term increases from O ( 1/ √ m ) to O ( 1 ) . At this upper extreme , the bound becomes trivial , as such a constant upper bound would directly follow from just theO ( 1 ) bounds assumed on Y . On the other hand , as the widths decrease , the fidelity terms ( the second and third terms ) would likely decrease – this is because the simple functions in Glocal would find it increasingly easier to approximate f on the shrinking neighborhoods . This trade-off is intuitive . A function f that is hardly amenable to being fit by local explanations would require extremely tiny neighborhoods for Glocal to locally approximate it ( i.e . make the MNF terms small ) . For example , in an extreme case , when the neighborhoodsNmirx are set be point masses at x , it is trivially easy to find gx′ ( · ) ∈ Glocal with no approximation error . Thus , the complexity term would be too large in this case , implying that a hard-to-interpret f results in bad generalization . On the other hand , when f is easier to interpret , then we ’ d expect it to be well-approximated by Glocal even with wider neighborhoods . This allows one to afford smaller values for both the complexity and MNF terms . In other words , an easy-to-interpret f enjoys better generalization guarantees . Caveats . Our bound has two limitations worth noting . First , for high-dimensional datasets ( like image datasets ) , practical choices of Nmirx can lead to almost no overlap between neighborhoods , rendering the bound trivial in practice . This potentially poor dimension-dependence is a caveat similarly shared by bounds for non-parametric local regression , whereby increasing d results in an exponential increase in the required sample size ( Fan , 1993 ; Fan & Gijbels , 1996 ) . Nevertheless , we show later in our experiments that for lower-dimensional datasets and for practical choices of Nmirx , there can be sufficient neighborhood overlap to achieve values of ρS that are o ( √ m ) . A second caveat is that the second quantity , MNF ( f , g ) , requires unlabeled test data to be computed , which may be limiting if one is interested in numerically computing this bound in practice . It is however possible to get a bound without this dependence , although only on the test error of g rather than f ( see Appendix Theorem 4 ) . Nevertheless , we believe that the above bound has theoretical value in how it establishes a connection between the interpretability of f and its generalization .

Summary: This paper presents two main theoretical results as its main contributions.  First, the authors provide a bound on the model generalization in terms its local explainability.  This bound relates the model generalization, the training accuracy, local explainability, and the complexity of the explanations.  Second the authors provide a bound on local explainability generalization.  This bound describes the relation between training local explainability and test time local explainability.  Both these bounds make concrete intuitive notions around local explainability generalization --- for instance, as we increase the size of the local explanation, generalization will decrease.  Last, the authors provide short empirical evaluation to demonstrate the properties they describe in their bounds appear in practice.