Provably efficient, succinct, and precise explanations

1 Introduction . Modern machine learning systems have access to unprecedented amounts of computational resources and data , enabling them to rapidly train sophisticated models . These models achieve remarkable performance on a wide range of tasks , but their success appears to come at a price : the complexity of these models , responsible for their expressivity and accuracy , makes their inner workings inscrutable to human beings , rendering them powerful but opaque blackboxes . As these blackboxes become central in mission-critical systems and their predictions increasingly relied upon in high-stakes decisions , there is a growing urgency to address their lack of interpretability [ DVK17 , Lip18 ] . There has therefore been a surge of interest in the problem of explaining the predictions of blackbox models : why did a model f assign an instance x the label f ( x ) ? While there are numerous possibilities for what qualifies as an explanation ( see e.g . [ SK10 , BSH+10 , SVZ14 , RSG16 , KL17 , LL17 , STY17 ] ) , in this work we consider an explanation to be a set of x ’ s features that in conjunction essentially determines f ( x ) [ RSG18 ] . Following terminology from complexity theory , we call such an explanation a certificate . We seek succinct and precise certificates . Intuitively , this means that we would like the set of features to be as small as possible , and for it to nonetheless be a sufficient explanation for f ( x ) with high probability ; more formally , we have the following definition : ∗Alphabetical order . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Definition 1 ( Succinct and precise certificates ) . Let f : { ±1 } d → { ±1 } be a classifier and x ∈ { ±1 } d be an instance . We say that a set C ⊆ [ d ] of features is a size-k ε-error certificate for x if both of the following hold : ◦ Succinctness : |C| ≤ k , ◦ Precision : Pr y∼ { ±1 } d [ f ( y ) 6= f ( x ) | yC = xC ] ≤ ε , where we write ‘ yC = xC ’ to mean that y and x agree on all the features in C. Our main result . We give an efficient certificate-finding algorithm with provable guarantees on the succinctness and precision of the certificates that it returns . Our algorithm is model agnostic , requiring no assumptions about the structure of f . Theorem 1 . Our algorithm A is given as input an instance x and parameters ε , δ ∈ ( 0 , 1 ) . It makes queries to a blackbox model f and returns a certificate A ( x ) for x with the following guarantees . With probability 1− δ over a uniform random instance x ∼ { ±1 } d , A ( x ) is an ε-error certificate of size poly ( C ( f ) , 1/ε , 1/δ ) , where C ( f ) is the “ average certificate complexity '' of f . The time and query complexity of A is poly ( d , C ( f ) , 1/ε , 1/δ ) . There is a sizable literature on certificate finding and related problems , studying them from both algorithmic and hardness perspectives . Previous algorithms , which we overview next , were either efficient but lacked provable guarantees on the succinctness and precision of the certificates that they return , or achieved such guarantees but were inefficient . Furthermore , our algorithm circumvents several hardness results for finding succinct and precise certificates , which we also discuss next . 1.1 Prior work on certificate finding . Efficient heuristics . Recent work of Ribeiro , Singh , and Guestrin [ RSG18 ] studies certificates ( which they term anchors ) from an empirical perspective . They demonstrate , through experiments and user studies , the effectiveness of certificates as explanations across a variety of domains and tasks . Their work highlights the ease of understanding of certificates by human beings and their clarity of scope . The authors also point out several advantages of certificates over LIME explanations [ RSG16 ] . Their work gives an efficient heuristic , based on greedy search , for finding high-precision certificates . However , there are no guarantees on the succinctness of these certificates , and in fact , it is easy to construct classifiers f : { ±1 } d → { ±1 } with near-minimal certificate complexity , C ( f ) = 2 , for which their heuristic returns certificates of near-maximal size , Ω ( d ) . ( We elaborate on this in the body of this paper . ) This should be contrasted with the guarantees of Theorem 1 ; specifically , the dimension-independent bound on the sizes of the certificates that our algorithm returns . Prime implicants . A separate line of work has focused on finding prime implicants [ Ign20 , DH20 , INM19 , INMS19 , SCD18 ] . In the terminology of Definition 1 , an implicant is a 0-error certificate , and an implicant is prime if its error increases whenever a single feature is removed from it . We note that an implicant being prime ( i.e . of minimal size ) is not equivalent to it being the most succinct ( i.e . of minimum size ) : a classifier f : { ±1 } d → { ±1 } can have an implicant of size 1 and also prime implicants of size d− 1 . While 0-error certificates are desirable for their perfect precision , it is often impossible to find them efficiently . With no assumptions on f , even verifying that a certificate has 0 error requires querying f on the potentially exponentially many possible instances consistent with that certificate . Existing algorithms therefore focus on specific model classes . For example , [ DH20 ] gives an algorithm for enumerating all prime implicants for a restricted class of circuits . Still , there are numerous hardness results even for model-specific algorithms . For CNF formulas , determining whether a certificate is 0-error is NP-complete . For general size-m circuits , finding the an 0-error certificate that has size within m1−ε of the smallest possible is NPNP-complete for any ε > 0 [ Uma01 ] . Setting aside the computational intractability of finding 0-error certificates , we note that they are in general much less succinct than ε-error certificates . It is easy to construct examples where a size-1 certificate with 0.01-error exists , but the only 0-error certificate is the trivial one of size d containing all features . In general , there is a natural tradeoff between the two desiderata of succinctness and precision , and our algorithm allows the user to choose their desired tradeoff rather than forcing them to choose perfect precision at the cost of succinctness . Hardness of finding approximate certificates . There are also intractability results for finding the smallest ε-error certificate . [ WMHK21 ] show that for any ε , determining whether there exists an ε-error certificate of size k for a given circuit and instance is NPPP-complete . Furthermore , they show that assuming P 6= NP , there is no efficient algorithm that can even approximate the size of the smallest ε-error certificate to within a factor of d1−α for any α > 0 . Theorem 1 circumvents these hardness results because our algorithm is only expected to succeed for most ( at least a 1− δ fraction ) rather than all instances , and only returns a small certificate relative to the average certificate complexity of the model , rather than smallest for a particular instance . 1.2 Our approach and techniques . We connect certificate finding to a new algorithmic problem that we introduce , that of implicitly learning decision trees . The key to this connection is a deep result from complexity theory , Smyth ’ s theorem [ Smy02 ] , which enables us to relate the certificate and decision tree complexities of functions . We then show how recently developed decision tree learning algorithms [ BLT20b , BGLT20 ] can be extended to solve the implicit learning problem . In more detail , there are three modular components to our approach : ◦ Implicitly learning decision trees . Decision trees are the canonical example of an interpretable model . Their predictions admit simple explanations : a certificate for an instance x is the root-to-leaf path in the tree that x follows . A natural and wellstudied approach to explaining a blackbox model f is therefore to first learn a decision tree T that well-approximates f , and with this surrogate decision tree T in hand , one can then output a certificate for any instance x by returning the corresponding path in T [ CS95 , BS96 , VAB07 , ZH16 , VLJ+17 , BKB17 , VS20 ] . A limitation of this approach lies in the fact that many models of interest are inherently complex and can not be wellapproximated by a decision tree of tractable size . To circumvent this , we introduce the problem of implicitly learning decision trees . Roughly speaking , an algorithm for this task allows one to efficiently navigate a surrogate decision tree T for f—without building T in full . With such an algorithm , the complexity of finding a certificate scales with the depth of T , making it exponentially more efficient than building T in full , the complexity of which scales with its overall size . ◦ Relating certificate and decision tree complexities . To translate algorithms for implicitly learning decision trees into algorithms for finding certificates , we apply a theorem of Smyth that relates the certificate complexity of a function to its decision tree complexity . The notion of certificates is central to complexity theory , where it is basis of the complexity class NP . ( Smyth ’ s result resolved a longstanding conjecture of Tardos [ Tar89 ] that was motivated by the relationship between P and NP ∩ coNP . ) ◦ An efficient algorithm with provable guarantees . With the two items above in hand , we are able to leverage recent advances in decision tree learning to design certificate-finding algorithms . Specifically , we show that the decision tree learning algorithm of Blanc , Gupta , Lange , and Tan [ BGLT20 ] can be extended to the setting of implicit learning . Our resulting certificate-finding algorithm is simple : it constructs a certificate C for an instance x by recursively adding to C the most noise stabilizing feature . Fruitful connections between noise stability and learnability have long been known [ KOS04 , KKMS08 ] ; our work further demonstrates its utility for certificate finding . Our overall approach falls within the framework of Local Computation Algorithms [ RTVX11 ] . Such algorithms solve computational problems for which the output—in our case , the surrogate decision tree—is so large that returning it in its entirety would be intractable . Local computation algorithms are able access and return only select parts of the output—in our case , the path through the tree that corresponds to the specific instance x of interest—efficiently and consistently . 1.3 Discussion of broader context : global and local explanations . Existing approaches to post-hoc explanations mostly fall into two categories . Global explanations seek to capture the behavior of the entire model f , often by approximating it with a simple and interpretable model such as a decision tree [ CS95 , BS96 , VAB07 , ZH16 , VLJ+17 , BKB17 , VS20 ] or a set of rules [ LKCL19 , LAB20 ] . A limitation of such approaches , alluded to above and also discussed in numerous prior works ( see e.g . [ RSG16 , RSG18 ] ) , is that complex models often can not be well-approximated by simple ones . In other words , using a simple surrogate model necessarily results in low fidelity to the original model . Our work falls in the second category of local explanations [ SK10 , BSH+10 , SVZ14 , RSG16 , KL17 , LL17 , RSG18 ] . These seek to explain f ’ s label for specific instances x . Several of these approaches are based on notions of f being “ simple around x ” : for example , LIME explanations [ RSG16 ] show that f is “ approximately linear around x ” , and certificates , the focus of our work , show that f is “ approximately constant in a subspace containing x ” . The corresponding algorithms can therefore be run on models that are too complex to be faithfully represented by a simple global surrogate model . While our work falls in the category of local explanations , we believe that our new approach of “ explaining by implicit learning ” enjoys advantages of both local and global methods . The local explanations that our algorithm returns are all consistent with a single decision tree T that wellapproximates f globally . The implicit nature of the learning task allows for T to be intractably large , hence allowing for corresponding algorithms to be run on complex models f , circumventing the limitation of global methods discussed above . On the other hand , the existence of a single global surrogate decision tree , albeit one that may be too large to construct in full , affords several advantages of global methods . We list a few examples : ◦ Partial information about global structure . Our implicit learning algorithm can efficiently construct the subgraph of T comprising the root-to-leaf paths of a few specific instances ; alternatively , it can construct the subtree of T rooted at a certain node , or the first few layers of T . All of these could shed light on the global behavior of f . ◦ Feature importance information . Our implicit decision tree T has useful properties beyond being a good approximator of f . As we will show , its structure carries valuable semantic information about f , since the feature queried at any internal node v is the most “ noise stabilizing ” feature of the subfunction fv . The features in the certificates that our algorithm returns can be ordered accordingly , each being the most noise stabilizing feature of the subfunction determined by x ’ s value on the previous features . ◦ Measures of similarity between instances . Every decision tree T naturally induces a “ similarity distance ” between instances , given by the depth of their lowest common ancestor within T . Therefore pairs of instances that share a long common path in T before diverging ( or do not diverge at all ) are considered very similar , whereas pairs of instances that diverge early on , say at the root , are considered very dissimilar . ( Such tree-based distance functions have been influential in the study of hierarchical clustering ; see e.g . [ Das16 ] . ) This distance between two instances can be easily calculated from the certificates that our algorithm returns for them .

Summary: The paper considers a new algorithm for creating  explanations of predictions of arbitrary black box models.  In this paper, an explanation is a subset of the input features that explain the prediction in the sense that changing the input features, not in the explanation subset at random, has only a small probability of changing the model prediction.  The paper considers only binary features and binary classification and assumes uniform distribution over the input.  The main result is an algorithm that with probability 1-delta over the random choice of input provides an explanation (subset of features) that is polynomial in the average size of the smallest explanation when averaged over all inputs (and 1/delta, 1/epsilon where epsilon is error allowed).  The main algorithm assumes an existing decision tree algorithms, and simply builds the path from the root to the leaf where the input data point would end up, the explanation being the features queried in the nodes on the way. This is possible due to the assumption of uniform distribution over the binary features.  The main component of the theorem is based on a result from complexity theory, that bound certificate complexity (an explanation is a certificate) of a function, polynomially in terms of the decision tree complexity (depth) of the function, which leads to the polynomial in the bound given.  The second important part is the decision tree construction algorithm required for the explanation algorithm. This methods is based on recent results on decision trees and that works for only uniform distribution over binary features and binary classes, providing the limiting features of the result shown. The algorithm is sped up with random sampling which is straightforward by the same assumptions.