Provably efficient, succinct, and precise explanations

Abstract: Guy Blanc, Jane Lange, Li-Yang Tan

Reviews and author responses:

This work proposes (1) a new theoretical model for algorithms to explain the
predictions of black-box classifiers and (2) an algorithm in this model that
produces such explanations with polynomial guarantees on its running time for
producing explanations of a given fidelity.

The model is an analogue of learning under the uniform distribution, making use
of the ability to query the black-box classifier. Given a point of interest,
the algorithm finds a conjunction that is relatively small with high probability
(over the draw of a point of interest) and such that points matching the
conjunction receive the same label as the point of interest with high
probability over subsequent draws of points. All draws are w.r.t. the uniform
distribution on Boolean features. Note that the size of the conjunction 
therefore also corresponds directly to its coverage, which is 1/2^size.

The algorithm is an adaptation of recent decision tree learning algorithms,
that greedily branch on the attribute that reduces the noise sensitivity the 
most. The explanation is given by the branch that the point of interest follows
in this tree. The guarantees follow from a connection between certificate 
complexity and decision tree depth.
  Of the local explanation methods that have been proposed, I feel that anchors
were one of the better motivated options. So, it's interesting to propose a
model that captures something like anchors, in which an algorithm can provide 
guarantees, which is what this work does. The algorithm itself is pretty simple
and natural. I appreciate this.

The one obvious weakness of this model is that it is restricted to the uniform 
distribution on attributes. The paper suggests that it can be generalized, and
indeed perhaps it can be generalized to product distribution, but this is still
pretty limited. As a point of context, there is work (not referenced here) that
considered an analogous model in a "distribution-free" (arbitrary distribution)
setting:

Juba and Li, More accurate learning of k-DNF reference classes, AAAI 2020.

That work showed that in a distribution-free model, algorithms for finding
such conjunctive explanations would give algorithms for learning DNF in the
usual supervised learning model. Indeed, the reduction is pretty simple -- the
explanation-finding algorithm gives weak learners for DNF -- and so it seems
that these conjunctive explanations can only be found in models in which 
learning DNF is tractable. To date, this means membership query algorithms on
the uniform distribution and product distributions with some caveats. 
Fortunately, this is a setting where the membership queries are not an obstacle
at all since we assume we can query the classifier on arbitrary points, but of
course the distribution restrictions are a serious obstacle. In any case, these
negative results temper our expectations: there was not much scope for the work
to accomplish much more than it does, given the state of the art.

The one thing that I think would have made this work significantly stronger
would have been experiments. Ribeiro et al. have made the code for their 
experiments available, and it probably would not have been too difficult to
implement this algorithm. Even if the guarantee falls short of capturing the
real world, there's still a definite possibility that we would see an empirical
advantage to a better-motivated search algorithm than that used by Ribeiro et 
al.

I hereby acknowledge that I read the authors' response and it did not detract
from my original opinion of the paper. As mentioned above, I am skeptical that the setting can be generalized too much
further than the one considered here. Otherwise, this was discussed well.


The paper discusses the problem of providing explanations when predicting that some instance x belongs to a particular class. In this direction the authors use a decision tree in order to justify the assigned label, but the interesting part is that the decision tree is only implicitly learnt. Basically such an explanation is provided as a list of variables that one should query on the (implicit) decision tree, from the root node down to a leaf node that provides the classification for x. Furthermore, such an explanation can act as a certificate in the sense that indeed the variables mentioned in the explanation, have low error rate with high probability, when the attributes that we find along this path remain the same as in the query instance x, for any point sampled from the underlying distribution.
  Provide a full review of the submission, including its originality, quality, clarity, and significance. See https://neurips.cc/Conferences/2021/Reviewer-Guidelines for guidance on questions to address in your review, and /faq for how to incorporate Markdown and LaTeX into your review.

I think that the paper is looking at an interesting problem and with an interesting point of view. In several situations the message is clear, but there are other situations that I find a little bit hard to grasp/accept. For example, when discussing about global and local explanations, in lines 137-138 the authors have the statement "...the existence of a single global surrogate decision tree, albeit one that may be too large to construct in full, affords several advantages of global methods." I think that this is an odd statement. 

Q1. Why would we not construct the tree that we want to learn fully? If the tree is too large to be constructed fully, then perhaps the dataset is too large?  

Getting along, I think that near Definition 4 a figure is missing. Or, at least, it would be nice to have a figure there, or in the appendix, and also explain what is discussed. Is alpha (which is used to refer to a node) a set? It looks like that from the comment in lines 192-193. Similarly, in step 3c of the algorithm in Figure 1, alpha is obtained as a set of the queries that we have encountered in the root-to-node path. Then, one question is:

Q2. How do we know which one of the variables we need to query for that particular node? Also, since we are at it, every time we have a node in a root-to-leaf path, we include the query of the node, regardless if the answer is "yes" or "no". In other words, we keep track of the coordinate that is being queried at the node, but somehow we ignore the answer to the question asked, which subsequently determines which path we will follow further below?

Finally, the authors rely on implicitly learning a decision tree by revisiting a recent algorithm for decision trees that can sustain noise. The interesting thing is that the authors claim exponential efficiency gains, simply because they work in an implicit setting. This is a statement that I find hard to grasp. 

Q3. If this is such a beneficial approach (of implicitly learning a tree), then why not always be doing that? In which direction do we obtain exponential efficiency gains? Are there any interesting remarks connecting such gains with results that we can obtain from standard VC theory for decision trees (even if the comment has to be on gains that we obtain under a particular distribution)?

Q4. In Definition 9, why does the tree have to be complete?

Remarks

R1. In Definition 6, I think p should be 1/2, since you are working under the uniform distribution?

R2. Line 338 -> expression?

*After the rebuttal:*

My opinion for the paper has improved after reading the reviews of others and the response of the author(s). Therefore I am increasing my rating from 5 to 6.  I am satisfied with the discussion of the authors.

The submission proposes a new theoretical algorithm for computing certificates that can explain the behavior of a blackbox model f on an input data point x. The core feature of the algorithm is that it provides strong theoretical guarantees: it can be made to work correctly with high probability, produce a highly accurate certificate while running in time polynomial in the "average" certificate complexity of f.  The submission as a whole, and the introduction in particular, are very well written (in spite of a few typos here and there). There seems to have been a real effort put into the submission to make at least the high-level ideas accessible to non-experts. The core result is non-trivial and relies on several components: (1) the idea of learning only relevant parts of decision trees (which saves an exponential factor that separates the depth and size of a decision tree), (2) an adaptation of Smyth's result connecting global certificates with global decision trees that can be applied to the subtrees of the decision tree identified in step 1, and (3) an application of known results on noise-stabilizing features.

The result fits well in the scope of NeurIPS, and its technical contribution is sufficient to warrant publication. I could not verify the correctness of each individual claim made in the submission, but those that I could were correct. I recommend acceptance.

One specific point I am wondering about is the sentence on rows 81-82, which makes it sound as if Theorem 1 cannot be strengthened by dropping the "probably" part (i.e., the 1-delta probability of success) due to existing lower bounds. However, at least to me it is not clear why or whether this should be the case, since Theorem 1 also depends polynomially on the "average certificate complexity". If the lower bounds mentioned on rows 76-80 have small "average certificate complexity" then please add a remark or explanation about this; otherwise please weaken or reformulate the sentence. (If possible, I would be interested in a response concerning this point.)

Minor comments:
-row 32: "certificates the it returns" -> "certificates that it returns"
-row 91 and elsewhere: "an certificate" should be "a certificate"
-row 97: "in fact" should be "in the fact"
-row 110: "we *are* able"

Edit: I acknowledge having read the rebuttal and thank the authors for the clarifications. Yes.

 >The submission as a whole, and the introduction in particular, are very well written (in spite of a few typos here and there). There seems to have been a real effort put into the submission to make at least the high-level ideas accessible to non-experts. The core result is non-trivial and relies on several components

Thanks for these kind words!

>One specific point I am wondering about is the sentence on rows 81-82, which makes it sound as if Theorem 1 cannot be strengthened by dropping the "probably" part. 

Great catch! The work we cite only proves that it is hard to find a certificate for a particular instance relative to the optimal certificate for that instance. Our certificate size depends on the average certificate complexity and so is incomparable. In the next revision, we will change lines 81-82 to: “Theorem 1 circumvents these hardness results because our algorithm is only expected to succeed for most (at least a $1 − \delta$ fraction) rather than all instances, and only return a small certificate relative to the average certificate complexity of the model, rather than smallest for a particular instance.”

>Minor comments...: 

Thanks! These are very helpful.

 >Q1. Why would we not construct the tree that we want to learn fully? If the tree is too large to be constructed fully, then perhaps the dataset is too large?

An implicit decision tree allows us to construct a single path of the tree without building the whole tree. A tree of depth $d$ can have $2^d$ leaves. Even if we have enough resources to explicitly learn the entire tree in time $2^d$, constructing just a single path in time $\mathrm{poly}(d)$ is exponentially more efficient (and is sufficient for the intended application of explaining a specific instance $x$).

As is standard in the literature on post-hoc explanations, we anticipate a practitioner first using their favorite high-performing and black-box model, such as a neural network, to fit the data. Then, they use our implicit learning framework to explain that model. Even when the data set is small, an intractably large decision tree may be required to approximate that black-box model. Our approach of implicit learning makes this tractable.

Furthermore, we note that our approach achieves exponential savings not just in time and memory complexity but also in query complexity, which is important when the black-box model is expensive to evaluate.

> Q2. How do we know which one of the variables we need to query for that particular node?

In definition 4, we require the implicit decision tree to implement the “query” operation. This operation returns which variable to query at a particular node. We encode which node we are at through alpha, which is a set of pairs of (coordinate, $x$’s value on that coordinate), where $x$ is the input we seek to explain.

>In other words, we keep track of the coordinate that is being queried at the node, but somehow we ignore the answer to the question asked, which subsequently determines which path we will follow further below?

Whenever we add a coordinate to $\alpha$, we also add $x$’s value on that coordinate. In step (3c) of Figure 1, adding ($i = x_i$) indicates adding coordinate $i$ with the value of $x$ on that coordinate to $\alpha$.

>Q3. If this is such a beneficial approach (of implicitly learning a tree), then why not always be doing that? In which direction do we obtain exponential efficiency gains? Are there any interesting remarks connecting such gains with results that we can obtain from standard VC theory for decision trees

The exponential gains stem from the fact that the depth of a tree, call it $d$, is generally exponentially smaller than its overall size, $2^d$.  For our application to explaining black-boxes, we only need to construct a particular path of the decision tree, the one corresponding to the instance $x$ that we seek to explain. The running time for building a single path through implicit decision trees scales with the depth of the path. However, if you want to learn the entire decision tree, the time scales with the number of paths in the tree, which is exponentially larger.
R.e. VC theory: in our setting, queries to the model are allowed and our goal is to construct a small part of the hypothesis. In VC theory, only random examples from the model are allowed and the goal is to learn the entire hypothesis.

>Q4. In Definition 9, why does the tree have to be complete?

Our Theorem 3 is from BLT20b and applies to the particular tree in Definition 9, so we used the same definition as they did. We believe they chose to make the tree complete just to simplify their analysis.

>R1. In Definition 6, I think p should be 1/2, since you are working under the uniform distribution?

$p$ is the noise-rate, which is not the same as the distribution of our input. We draw $x$ uniformly at random, so it is roughly half zeros and half ones. Then, we re-randomize a $p$ fraction of the coordinates of $x$ to create $x’$. In Theorem 3, we set p to a value that is not $1/2$ (it is smaller).

>R2. Line 338 -> expression?

Thanks for pointing that out!


 >Of the local explanation methods that have been proposed, I feel that anchors were one of the better motivated options. So, it's interesting to propose a model that captures something like anchors, in which an algorithm can provide guarantees, which is what this work does. The algorithm itself is pretty simple and natural. I appreciate this.

Thank you for these kind words!

>It seems that these conjunctive explanations can only be found in models in which learning DNF is tractable … there was not much scope for the work to accomplish much more than it does, given the state of the art

We agree, though we are optimistic that there are concrete next steps to explore. As best we can tell, the Juba and Li paper makes an incomparable assumption about the model than we do. Their lower bound assumes the model is computed by a k-DNF, corresponding to *every* instance labeled as 1 having a succinct certificate. In contrast, we assume that *most*, not all, inputs have a succinct certificate regardless of whether they are labeled as 0 or 1.

>The one obvious weakness of this model is that it is restricted to the uniform distribution on attributes.

Indeed our provable guarantees require feature and distributional assumptions; however, a core aspect of our framework is completely general: the reduction from certificate-finding to implicitly learning decision trees (all of section 2; in particular Lemma 2.1). We also note that there is a distribution-free algorithm for (non-implicit) learning of decision trees that runs in quasipolynomial time (Ehrenfeucht and Haussler, Learning decision trees from random examples, Information and Computing 89); however, it is not known to support the implicit-learning operations as efficiently as we require in this work. Finding implicit-learning algorithms for the distribution-free setting, based on [EH89] or otherwise, is an exciting direction for followup work.

>The one thing that I think would have made this work significantly stronger would have been experiments.

We agree. The goal of our work is to design a polynomial-time algorithm whose certificates are provably succinct and precise. An in-depth experimental comparison of our algorithm to alternatives is a concrete direction for followup work.


Meta-review: This paper relates the problem of interpreting the output of a black-box machine learning classifier at any point x by releasing an (exponentially small) fragment of a decision tree -- the conjunction labeling a path from root to leaf activated at x. This is natural if one views an "explanation" as a "certificate" for the value of a function at a given point (and ties in neatly to certificate complexity studied in computational complexity theory). In order to accomplish construct such explanations efficiently, the paper introduces and gives an algorithm for the problem of implicitly learning decision trees based on some recent developments in the area. 

Interpreting machine learning classifiers is an important goal and one that could indeed benefit from fresh theoretical approaches. This innovative paper introduces one approach that could fit the bill. We recommend acceptance. 

Decision: accept