Foundations of Symbolic Languages for Model Interpretability

1 Introduction . Context . The degree of interpretability of a machine learning ( ML ) model seems to be intimately related with the ability to “ answer questions ” about it . Those questions can either be global ( behavior of the model as a whole ) or local ( behavior regarding certain instances/features ) . Concrete examples of such questions can be found in the recent literature , including , e.g. , queries based on “ anchors ” , which are parts of an instance that are sufficient to justify its classification [ 4 , 13 , 16 , 32 ] , and numerical scores that measure the impact of the different features of an instance on its result [ 24 , 31 , 36 ] . It is by now clear that ML interpretability admits no silver-bullet [ 18 ] , and that in many cases a combination of different queries may be the most effective way to understand a model ’ s behavior . Also , model interpretability takes different flavors depending on the application domain one deals with . This naturally brings to the picture the need for general-purpose specification languages that can provide flexibility and expressiveness to practitioners specifying interpretability queries . An even more advanced requirement for these languages is to be relatively easy to use in practice . This tackles the growing need for bringing interpretability methods closer to users with different levels of expertise . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . One way in which these requirements can be approached in a principled way is by developing a declarative interpretability language , i.e. , one in which users directly express the queries they want to apply in the interpretability process ( and not how these queries will be evaluated ) . This is of course reminiscent of the path many other areas in computer science have followed , in particular by using languages rooted in formal logic ; so has been the case , e.g. , in data management [ 1 ] , knowledge representation [ 3 ] , and model checking [ 15 ] . One of the advantages of this approach is that logics have a well-defined syntax and clear semantics . On the one hand , this ensures that the obtained explanations are provably sound and faithful to the model , which avoids a significant drawback of several techniques for explaining models in which the explanations can be inaccurate , or require themselves to be further explained [ 33 ] . On the other hand , a logical root facilitates the theoretical study of the computational cost of evaluation and optimization for the queries in the language . Our proposal . Our first contribution is the proposal of a logical language , called FOIL , in which many simple yet relevant interpretability queries can be expressed . We believe that FOIL can further serve as a basis over which more expressive interpretability languages can be built , and we propose concrete directions of research towards its expansion . In a nutshell , given a decision modelM that performs classification over instances e of dimension n , FOIL can express properties over the set of all partial and full instances of dimension n. A partial instance e is a vector of dimension n in which some features are undefined . Such undefined features take a distinguished value ⊥ . An instance is full if none of its features is undefined . The logic FOIL is simply first-order logic with access to two predicates on the set of all instances ( partial or full ) of dimension n : A unary predicate POS ( e ) , stating that e is a full instance thatM classifies as positive , and a binary predicate e ⊆ e′ , stating that instance e′ potentially fills some of the undefined features of instance e ; e.g. , ( 1 , 0 , ⊥ ) ⊆ ( 1 , 0 , 1 ) , but ( 1 , 0 , ⊥ ) 6⊆ ( 1 , 1 , 1 ) . As an overview of our proposal , consider the case of a bank using a binary model to judge applications for loans . Figure 1a illustrates the problem with concrete features , and Figure 1b presents an example of a concrete interactive syntax . In Figure 1b , after loading and exploring the model , the interaction asks whether the model could give a loan to a person who is married and does not have kids . Assuming that the “ Accepted ” class is the positive one , this interaction can easily be formalized in FOIL by means of the query ∃x ( POS ( x ) ∧ ( ⊥ , ⊥ , ⊥ , ⊥ , 0 , 1 , ⊥ ) ⊆ x ) . Theoretical contributions . The evaluation problem for a fixed FOIL query ϕ is as follows . Given a decision modelM , is it true that ϕ is satisfied under the interpretation of predicates ⊆ and POS defined above ? An important caveat about this problem is that , in order to evaluate ϕ , we need to potentially look for an exponential number of instances , even if the features are Boolean , thus rendering the complexity of the problem infeasible in some cases . Think , for instance , of the query ∃x POS ( x ) , which asks ifM has at least one positive instance . Then this query is intractable for every class of models for which this problem is intractable ; e.g. , for the class of propositional formulas in CNF ( notice that this is nothing but the satisfiability problem for the class at hand ) . The main theoretical contribution of our paper is an in-depth study of the computational cost of FOIL on two classes of Boolean models that are often deemed to be “ easy to interpret ” : decision trees and ordered binary decision diagrams ( OBDDs ) [ 10 , 14 , 19 , 28 , 33 ] . An immediate advantage of these models over , say , CNF formulas , is that the satisfiability problem for them can be solved in polynomial time ; i.e. , the problem of evaluating the query ∃xPOS ( x ) is tractable . Our study aims to ( a ) “ measure ” the degree of interpretability of said models in terms of the formal yardstick defined by the language FOIL ; and ( b ) shed light on when and how some simple interpretability queries can be evaluated efficiently on these decision models . We start by showing that , in spite of the aforementioned claims on the good level of interpretability for the models considered , there is a simple query in FOIL that is intractable over them . In fact , such an intractable query has a natural “ interpretability ” flavor , and thus we believe this proof to be of independent interest . However , these intractability results should not immediately rule out the use of FOIL in practice . In fact , it is well known that a logic can be intractable in general , but become tractable in practically relevant cases . Such cases can be obtained by either restricting the syntactic fragment of the logic considered , or the structure of the models in which the logic is evaluated . We obtain positive results in both directions for the models we mentioned above . We explain them next . Syntactic fragments . We show that queries in ∃FOIL , the existential fragment of FOIL , admit tractable evaluation over the models we study . However , this language lacks expressive power for capturing some interpretability queries of practical interest . We then introduce ∃FOIL+ , an extension of ∃FOIL with a finite set of unary universal queries from FOIL that is enough for expressing some relevant interpretability queries . We provide a characterization theorem for the tractability of ∃FOIL+ over any class of Boolean decision models that reduces the tractability of this fragment to the tractability of two fixed and specific FOIL queries . Then we prove that said queries are tractable over perceptrons , which implies the tractability of ∃FOIL+ for this model . Unfortunately , the evaluation of said queries is NP-hard for decision trees and OBDDs . Both the proof of tractability for perceptrons and intractability for decision trees and OBDDs are relatively simple , thus showing that the characterization theorem provides a useful technique for understanding which models are tractable for the evaluation of ∃FOIL+ . Structural restrictions . We restrict the models allowed in order to obtain tractability of evaluation for arbitrary FOIL queries . In particular , we show that evaluation of ϕ , for ϕ a fixed FOIL query , can be solved in polynomial time over the class of OBDDs as long as they are complete , i.e. , any path from the root to a leaf of the OBDD tests every feature from the input , and have bounded width , i.e. , there is a constant bound on the number of nodes of the OBDD in which a feature can appear . Practical implementation . We designed FOIL with a minimal set of logical constructs and tailored for models with binary input features . These decisions are reasonable for a detailed theoretical analysis but may hamper FOIL usage in more general scenarios , in particular when models have ( many ) categorical or numerical input features , and queries are manually written by non-expert users . To tackle this we introduce a more user-friendly language with a high-level syntax ( à la SQL in the spirit of the query in Figure 1b ) that can be compiled into FOIL queries . Moreover , we present a prototype implementation that can be used to query decision trees trained in standard ML libraries by binarizing them into models ( a subclass of binary decision diagrams ) over which FOIL queries can be efficiently evaluated . We also test the performance of our implementation over synthetic and real data giving evidence of the usability of FOIL as a base for practical interpretabilty languages . 2 A Logic for Interpretability Queries . Background . An instance of dimension n , with n ≥ 1 , is a tuple e ∈ { 0 , 1 } n. We use notation e [ i ] to refer to the i-th component of this tuple , or equivalently , its i-th feature . Moreover , we consider an abstract notion of a model of dimension n , and we define it as a Boolean function M : { 0 , 1 } n → { 0 , 1 } . That is , M assigns a Boolean value to each instance of dimension n , so that we focus on binary classifiers with Boolean input features . Restricting inputs and outputs to be Boolean makes our setting cleaner while still covering several relevant practical scenarios . We use notation dim ( M ) for the dimension of a modelM . A partial instance of dimension n is a tuple e ∈ { 0 , 1 , ⊥ } n. Intuitively , if e [ i ] = ⊥ , then the value of the i-th feature is undefined . Notice that an instance is a particular case of a partial instance where all features are assigned value either 0 or 1 . Given two partial instances e1 , e2 of dimension n , we say that e1 is subsumed by e2 if for every i ∈ { 1 , . . . , n } such that e1 [ i ] 6= ⊥ , it holds that e1 [ i ] = e2 [ i ] . That is , e1 is subsumed by e2 if it is possible to obtain e2 from e1 by replacing some unknown values . Notice that a partial instance e can be thought of as a compact representation of the set of instances e′ such that e is subsumed by e′ , where such instances e′ are called the completions of e. Models . A binary decision diagram ( BDD [ 38 ] ) over instances of dimension n is a rooted directed acyclic graphM with labels on edges and nodes such that : ( i ) each leaf is labeled with true or false ; ( ii ) each internal node ( a node that is not a leaf ) is labeled with a feature i ∈ { 1 , . . . , n } ; and ( iii ) each internal node has two outgoing edges , one labeled 1 and the another one labeled 0 . Every instance e ∈ { 0 , 1 } n defines a unique path πe = u1 · · ·uk from the root u1 to a leaf uk ofM such that : if the label of ui is j ∈ { 1 , . . . , n } , where i ∈ { 1 , . . . , k − 1 } , then the edge from ui to ui+1 is labeled with e [ j ] . Moreover , the instance e is positive , denoted byM ( e ) = 1 , if the label of uk is true ; otherwise the instance e is negative , which is denoted byM ( e ) = 0 . A binary decision diagramM is free ( FBDD ) if for every path from the root to a leaf , no two nodes on that path have the same label . Besides , M is ordered ( OBDD ) if there exists a linear order < on the set { 1 , . . . , n } of features such that , if a node u appears before a node v in some path inM from the root to a leaf , then u is labeled with i and v is labeled with j for features i , j such that i < j . A decision tree is simply an FBDD whose underlying DAG is a tree . Finally , a perceptronM of dimension n is a pair ( w , t ) where w ∈ Rn and t ∈ R , and the classification of an instance e ∈ { 0 , 1 } n is defined as M ( e ) = 1 if and only if w · e ≥ t. In this paper , we focus on the following classes of models : OBDD , the class of ordered BDDs , DTree , the class of decision trees , and Ptron , the class of perceptrons . None of these classes directly subsume the other : decision trees are not necessarily ordered , while the underlying DAG of an OBDD is not necessarily a tree . In fact , it is known that neither OBDDs can be compiled into polynomial-size decision trees nor decision trees into polynomial-size OBDDs [ 6 , 17 ] . Perceptrons on the other hand can only model linear decision boundaries and thus are inherently less expressive than decision trees or OBDDs . It is also known that perceptrons can not be compiled in polynomial time to decision trees or OBDDs unless P = NP [ 4 ] . The logic FOIL . We consider first-order logic over a vocabulary consisting of a unary predicate POS and a binary predicate ⊆ . This logic is called first-order interpretability logic ( FOIL ) , and it is our reference language for defining conditions on models that we would like to reason about . In particular , predicate POS is used to indicate the value of an instance in a model , while predicate ⊆ is used to represent the subsumption relation among partial instances . In what follows , we show that many natural properties can be expressed in a simple way in FOIL , demonstrating the suitability of this language for the purpose of expressing explainability queries . We assume familiarity with the syntax and semantics of first-order logic ( see the appendix for a review of these concepts ) . In particular , given a vocabulary σ consisting of relations R1 , . . . , R ` , recall that a structure A over σ consists of a domain , where quantifiers are instantiated , and an interpretation for each relation Ri . Moreover , given a first-order formula ϕ defined over the vocabulary σ , we write ϕ ( x1 , . . . , xk ) to indicate that { x1 , . . . , xk } is the set of free variables of ϕ . Finally , given a structure A over the vocabulary σ and elements a1 , . . . , ak in the domain of A , we use A |= ϕ ( a1 , . . . , ak ) to indicate that formula ϕ is satisfied by A when each variable xi is replaced by element ai ( 1 ≤ i ≤ k ) . Our goal when introducing FOIL is to have a logic that allows to specify natural properties of models in a simple way . In this sense , we still need to define when a modelM satisfies a formula in FOIL , as M is not a structure over the vocabulary { POS , ⊆ } ( so we can not directly use the notion of satisfaction of a formula by a structure ) . More precisely , assuming that dim ( M ) = n , the structure AM associated toM is defined as follows . The domain of AM is the set { 0 , 1 , ⊥ } n of all partial instances of dimension n. An instance e ∈ { 0 , 1 } n is in the interpretation of predicate POS in AM if and only ifM ( e ) = 1 . Finally , a pair ( e1 , e2 ) is in the interpretation of predicate ⊆ in AM if and only if e1 is subsumed by e2 . Then , given a formula ϕ ( x1 , . . . , xk ) in FOIL and partial instances e1 , . . . , ek of dimension n , modelM is said to satisfy ϕ ( e1 , . . . , ek ) , denoted byM |= ϕ ( e1 , . . . , ek ) , if and only if AM |= ϕ ( e1 , . . . , ek ) . Evaluation problem . FOIL is our main tool in trying to understand how interpretable is a class of models . In particular , the following is the main problem studied in this paper , given a class C of models and a formula ϕ ( x1 , . . . , xk ) in FOIL . Problem : EVAL ( ϕ , C ) Input : A modelM∈ C of dimension n , and partial instances e1 , . . . , ek of dimension n Output : YES , ifM |= ϕ ( e1 , . . . , ek ) , and NO otherwise For example , assume that CNF , DNF are the classes of models given as propositional formulae in CNF and DNF , respectively . If ϕ = ∃x POS ( x ) , then EVAL ( ϕ , CNF ) is NP-complete and EVAL ( ϕ , DNF ) can be solved in polynomial time , as such problems correspond to the satisfiability problems for the propositional formulae in CNF and DNF , respectively . Given a modelM , it is important to notice that the size of the structure AM can be exponential in the size ofM . Hence , AM is a theoretical construction needed to formally define the semantics of FOIL , but that should not be built when verifying in practice if a formula ϕ is satisfied byM . In fact , if we are aiming at finding tractable algorithms for FOIL-evaluation , then we need to design an algorithm that uses directly the encoding ofM as a model ( for example , as a binary decision tree ) rather than as a logical structure . In other words , in order to evaluate a query ϕ = ∃x POS ( x ) over a modelM of dimension n , one could certainly iterate over all 2n instances e ∈ { 0 , 1 } n and evaluate M ( e ) . This of course impractical for even small-dimensional data . Therefore , evaluating formulas without iterating over the entire space of ( partial ) instances is the main technical challenge behind the results presented in this paper .

Summary: This paper introduces a simple logic for examining classifiers. It shows 1) The logic allows defining, in particular, a predicate capturing (minimal) sufficient reason explanations (i.e., a set of feature values that ensure the classifier gives a certain classification) and a predicate indicating whether the classifier makes a decision using a protected set of features. 2) The most natural version of the logic allows defining a predicate that is NP-hard to evaluate for both decision tree and OBDD classifiers 3) A simple fragment of the logic is adequate to capture the formulas indicated in (1), and there are two formulas such that if these can be evaluated in polynomial time for a model class, then the entire fragment is polynomial time 4) The fragment in (3) is polynomial-time decidable for decision trees and OBDDs (on account of the indicated criterion) 5) A restricted kind of OBDD allows the more general logic to be decided in polynomial time.