Foundations of Symbolic Languages for Model Interpretability

1 Introduction . Context . The degree of interpretability of a machine learning ( ML ) model seems to be intimately related with the ability to “ answer questions ” about it . Those questions can either be global ( behavior of the model as a whole ) or local ( behavior regarding certain instances/features ) . Concrete examples of such questions can be found in the recent literature , including , e.g. , queries based on “ anchors ” , which are parts of an instance that are sufficient to justify its classification [ 4 , 13 , 16 , 32 ] , and numerical scores that measure the impact of the different features of an instance on its result [ 24 , 31 , 36 ] . It is by now clear that ML interpretability admits no silver-bullet [ 18 ] , and that in many cases a combination of different queries may be the most effective way to understand a model ’ s behavior . Also , model interpretability takes different flavors depending on the application domain one deals with . This naturally brings to the picture the need for general-purpose specification languages that can provide flexibility and expressiveness to practitioners specifying interpretability queries . An even more advanced requirement for these languages is to be relatively easy to use in practice . This tackles the growing need for bringing interpretability methods closer to users with different levels of expertise . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . One way in which these requirements can be approached in a principled way is by developing a declarative interpretability language , i.e. , one in which users directly express the queries they want to apply in the interpretability process ( and not how these queries will be evaluated ) . This is of course reminiscent of the path many other areas in computer science have followed , in particular by using languages rooted in formal logic ; so has been the case , e.g. , in data management [ 1 ] , knowledge representation [ 3 ] , and model checking [ 15 ] . One of the advantages of this approach is that logics have a well-defined syntax and clear semantics . On the one hand , this ensures that the obtained explanations are provably sound and faithful to the model , which avoids a significant drawback of several techniques for explaining models in which the explanations can be inaccurate , or require themselves to be further explained [ 33 ] . On the other hand , a logical root facilitates the theoretical study of the computational cost of evaluation and optimization for the queries in the language . Our proposal . Our first contribution is the proposal of a logical language , called FOIL , in which many simple yet relevant interpretability queries can be expressed . We believe that FOIL can further serve as a basis over which more expressive interpretability languages can be built , and we propose concrete directions of research towards its expansion . In a nutshell , given a decision modelM that performs classification over instances e of dimension n , FOIL can express properties over the set of all partial and full instances of dimension n. A partial instance e is a vector of dimension n in which some features are undefined . Such undefined features take a distinguished value ⊥ . An instance is full if none of its features is undefined . The logic FOIL is simply first-order logic with access to two predicates on the set of all instances ( partial or full ) of dimension n : A unary predicate POS ( e ) , stating that e is a full instance thatM classifies as positive , and a binary predicate e ⊆ e′ , stating that instance e′ potentially fills some of the undefined features of instance e ; e.g. , ( 1 , 0 , ⊥ ) ⊆ ( 1 , 0 , 1 ) , but ( 1 , 0 , ⊥ ) 6⊆ ( 1 , 1 , 1 ) . As an overview of our proposal , consider the case of a bank using a binary model to judge applications for loans . Figure 1a illustrates the problem with concrete features , and Figure 1b presents an example of a concrete interactive syntax . In Figure 1b , after loading and exploring the model , the interaction asks whether the model could give a loan to a person who is married and does not have kids . Assuming that the “ Accepted ” class is the positive one , this interaction can easily be formalized in FOIL by means of the query ∃x ( POS ( x ) ∧ ( ⊥ , ⊥ , ⊥ , ⊥ , 0 , 1 , ⊥ ) ⊆ x ) . Theoretical contributions . The evaluation problem for a fixed FOIL query ϕ is as follows . Given a decision modelM , is it true that ϕ is satisfied under the interpretation of predicates ⊆ and POS defined above ? An important caveat about this problem is that , in order to evaluate ϕ , we need to potentially look for an exponential number of instances , even if the features are Boolean , thus rendering the complexity of the problem infeasible in some cases . Think , for instance , of the query ∃x POS ( x ) , which asks ifM has at least one positive instance . Then this query is intractable for every class of models for which this problem is intractable ; e.g. , for the class of propositional formulas in CNF ( notice that this is nothing but the satisfiability problem for the class at hand ) . The main theoretical contribution of our paper is an in-depth study of the computational cost of FOIL on two classes of Boolean models that are often deemed to be “ easy to interpret ” : decision trees and ordered binary decision diagrams ( OBDDs ) [ 10 , 14 , 19 , 28 , 33 ] . An immediate advantage of these models over , say , CNF formulas , is that the satisfiability problem for them can be solved in polynomial time ; i.e. , the problem of evaluating the query ∃xPOS ( x ) is tractable . Our study aims to ( a ) “ measure ” the degree of interpretability of said models in terms of the formal yardstick defined by the language FOIL ; and ( b ) shed light on when and how some simple interpretability queries can be evaluated efficiently on these decision models . We start by showing that , in spite of the aforementioned claims on the good level of interpretability for the models considered , there is a simple query in FOIL that is intractable over them . In fact , such an intractable query has a natural “ interpretability ” flavor , and thus we believe this proof to be of independent interest . However , these intractability results should not immediately rule out the use of FOIL in practice . In fact , it is well known that a logic can be intractable in general , but become tractable in practically relevant cases . Such cases can be obtained by either restricting the syntactic fragment of the logic considered , or the structure of the models in which the logic is evaluated . We obtain positive results in both directions for the models we mentioned above . We explain them next . Syntactic fragments . We show that queries in ∃FOIL , the existential fragment of FOIL , admit tractable evaluation over the models we study . However , this language lacks expressive power for capturing some interpretability queries of practical interest . We then introduce ∃FOIL+ , an extension of ∃FOIL with a finite set of unary universal queries from FOIL that is enough for expressing some relevant interpretability queries . We provide a characterization theorem for the tractability of ∃FOIL+ over any class of Boolean decision models that reduces the tractability of this fragment to the tractability of two fixed and specific FOIL queries . Then we prove that said queries are tractable over perceptrons , which implies the tractability of ∃FOIL+ for this model . Unfortunately , the evaluation of said queries is NP-hard for decision trees and OBDDs . Both the proof of tractability for perceptrons and intractability for decision trees and OBDDs are relatively simple , thus showing that the characterization theorem provides a useful technique for understanding which models are tractable for the evaluation of ∃FOIL+ . Structural restrictions . We restrict the models allowed in order to obtain tractability of evaluation for arbitrary FOIL queries . In particular , we show that evaluation of ϕ , for ϕ a fixed FOIL query , can be solved in polynomial time over the class of OBDDs as long as they are complete , i.e. , any path from the root to a leaf of the OBDD tests every feature from the input , and have bounded width , i.e. , there is a constant bound on the number of nodes of the OBDD in which a feature can appear . Practical implementation . We designed FOIL with a minimal set of logical constructs and tailored for models with binary input features . These decisions are reasonable for a detailed theoretical analysis but may hamper FOIL usage in more general scenarios , in particular when models have ( many ) categorical or numerical input features , and queries are manually written by non-expert users . To tackle this we introduce a more user-friendly language with a high-level syntax ( à la SQL in the spirit of the query in Figure 1b ) that can be compiled into FOIL queries . Moreover , we present a prototype implementation that can be used to query decision trees trained in standard ML libraries by binarizing them into models ( a subclass of binary decision diagrams ) over which FOIL queries can be efficiently evaluated . We also test the performance of our implementation over synthetic and real data giving evidence of the usability of FOIL as a base for practical interpretabilty languages . 2 A Logic for Interpretability Queries . Background . An instance of dimension n , with n ≥ 1 , is a tuple e ∈ { 0 , 1 } n. We use notation e [ i ] to refer to the i-th component of this tuple , or equivalently , its i-th feature . Moreover , we consider an abstract notion of a model of dimension n , and we define it as a Boolean function M : { 0 , 1 } n → { 0 , 1 } . That is , M assigns a Boolean value to each instance of dimension n , so that we focus on binary classifiers with Boolean input features . Restricting inputs and outputs to be Boolean makes our setting cleaner while still covering several relevant practical scenarios . We use notation dim ( M ) for the dimension of a modelM . A partial instance of dimension n is a tuple e ∈ { 0 , 1 , ⊥ } n. Intuitively , if e [ i ] = ⊥ , then the value of the i-th feature is undefined . Notice that an instance is a particular case of a partial instance where all features are assigned value either 0 or 1 . Given two partial instances e1 , e2 of dimension n , we say that e1 is subsumed by e2 if for every i ∈ { 1 , . . . , n } such that e1 [ i ] 6= ⊥ , it holds that e1 [ i ] = e2 [ i ] . That is , e1 is subsumed by e2 if it is possible to obtain e2 from e1 by replacing some unknown values . Notice that a partial instance e can be thought of as a compact representation of the set of instances e′ such that e is subsumed by e′ , where such instances e′ are called the completions of e. Models . A binary decision diagram ( BDD [ 38 ] ) over instances of dimension n is a rooted directed acyclic graphM with labels on edges and nodes such that : ( i ) each leaf is labeled with true or false ; ( ii ) each internal node ( a node that is not a leaf ) is labeled with a feature i ∈ { 1 , . . . , n } ; and ( iii ) each internal node has two outgoing edges , one labeled 1 and the another one labeled 0 . Every instance e ∈ { 0 , 1 } n defines a unique path πe = u1 · · ·uk from the root u1 to a leaf uk ofM such that : if the label of ui is j ∈ { 1 , . . . , n } , where i ∈ { 1 , . . . , k − 1 } , then the edge from ui to ui+1 is labeled with e [ j ] . Moreover , the instance e is positive , denoted byM ( e ) = 1 , if the label of uk is true ; otherwise the instance e is negative , which is denoted byM ( e ) = 0 . A binary decision diagramM is free ( FBDD ) if for every path from the root to a leaf , no two nodes on that path have the same label . Besides , M is ordered ( OBDD ) if there exists a linear order < on the set { 1 , . . . , n } of features such that , if a node u appears before a node v in some path inM from the root to a leaf , then u is labeled with i and v is labeled with j for features i , j such that i < j . A decision tree is simply an FBDD whose underlying DAG is a tree . Finally , a perceptronM of dimension n is a pair ( w , t ) where w ∈ Rn and t ∈ R , and the classification of an instance e ∈ { 0 , 1 } n is defined as M ( e ) = 1 if and only if w · e ≥ t. In this paper , we focus on the following classes of models : OBDD , the class of ordered BDDs , DTree , the class of decision trees , and Ptron , the class of perceptrons . None of these classes directly subsume the other : decision trees are not necessarily ordered , while the underlying DAG of an OBDD is not necessarily a tree . In fact , it is known that neither OBDDs can be compiled into polynomial-size decision trees nor decision trees into polynomial-size OBDDs [ 6 , 17 ] . Perceptrons on the other hand can only model linear decision boundaries and thus are inherently less expressive than decision trees or OBDDs . It is also known that perceptrons can not be compiled in polynomial time to decision trees or OBDDs unless P = NP [ 4 ] . The logic FOIL . We consider first-order logic over a vocabulary consisting of a unary predicate POS and a binary predicate ⊆ . This logic is called first-order interpretability logic ( FOIL ) , and it is our reference language for defining conditions on models that we would like to reason about . In particular , predicate POS is used to indicate the value of an instance in a model , while predicate ⊆ is used to represent the subsumption relation among partial instances . In what follows , we show that many natural properties can be expressed in a simple way in FOIL , demonstrating the suitability of this language for the purpose of expressing explainability queries . We assume familiarity with the syntax and semantics of first-order logic ( see the appendix for a review of these concepts ) . In particular , given a vocabulary σ consisting of relations R1 , . . . , R ` , recall that a structure A over σ consists of a domain , where quantifiers are instantiated , and an interpretation for each relation Ri . Moreover , given a first-order formula ϕ defined over the vocabulary σ , we write ϕ ( x1 , . . . , xk ) to indicate that { x1 , . . . , xk } is the set of free variables of ϕ . Finally , given a structure A over the vocabulary σ and elements a1 , . . . , ak in the domain of A , we use A |= ϕ ( a1 , . . . , ak ) to indicate that formula ϕ is satisfied by A when each variable xi is replaced by element ai ( 1 ≤ i ≤ k ) . Our goal when introducing FOIL is to have a logic that allows to specify natural properties of models in a simple way . In this sense , we still need to define when a modelM satisfies a formula in FOIL , as M is not a structure over the vocabulary { POS , ⊆ } ( so we can not directly use the notion of satisfaction of a formula by a structure ) . More precisely , assuming that dim ( M ) = n , the structure AM associated toM is defined as follows . The domain of AM is the set { 0 , 1 , ⊥ } n of all partial instances of dimension n. An instance e ∈ { 0 , 1 } n is in the interpretation of predicate POS in AM if and only ifM ( e ) = 1 . Finally , a pair ( e1 , e2 ) is in the interpretation of predicate ⊆ in AM if and only if e1 is subsumed by e2 . Then , given a formula ϕ ( x1 , . . . , xk ) in FOIL and partial instances e1 , . . . , ek of dimension n , modelM is said to satisfy ϕ ( e1 , . . . , ek ) , denoted byM |= ϕ ( e1 , . . . , ek ) , if and only if AM |= ϕ ( e1 , . . . , ek ) . Evaluation problem . FOIL is our main tool in trying to understand how interpretable is a class of models . In particular , the following is the main problem studied in this paper , given a class C of models and a formula ϕ ( x1 , . . . , xk ) in FOIL . Problem : EVAL ( ϕ , C ) Input : A modelM∈ C of dimension n , and partial instances e1 , . . . , ek of dimension n Output : YES , ifM |= ϕ ( e1 , . . . , ek ) , and NO otherwise For example , assume that CNF , DNF are the classes of models given as propositional formulae in CNF and DNF , respectively . If ϕ = ∃x POS ( x ) , then EVAL ( ϕ , CNF ) is NP-complete and EVAL ( ϕ , DNF ) can be solved in polynomial time , as such problems correspond to the satisfiability problems for the propositional formulae in CNF and DNF , respectively . Given a modelM , it is important to notice that the size of the structure AM can be exponential in the size ofM . Hence , AM is a theoretical construction needed to formally define the semantics of FOIL , but that should not be built when verifying in practice if a formula ϕ is satisfied byM . In fact , if we are aiming at finding tractable algorithms for FOIL-evaluation , then we need to design an algorithm that uses directly the encoding ofM as a model ( for example , as a binary decision tree ) rather than as a logical structure . In other words , in order to evaluate a query ϕ = ∃x POS ( x ) over a modelM of dimension n , one could certainly iterate over all 2n instances e ∈ { 0 , 1 } n and evaluate M ( e ) . This of course impractical for even small-dimensional data . Therefore , evaluating formulas without iterating over the entire space of ( partial ) instances is the main technical challenge behind the results presented in this paper .

Summary: The authors propose a declarative, logic-based language to query machine learning models. The language is tailored to a syntax and semantics that supports queries that explore the behaviour of the machine learning model. For instance, using Pos(e) as an atom that expresses that input e (which might contain undefined arguments) is a positive example, that is, an example classified as positive by the ML model, and using predicates for individual features such as “married” and “has kids”, one could query whether the model under consideration would give a loan to a person who is married and does not have kids.  Given such a logical language, it is natural to ask what the complexity of answering queries in the languages are and to restrict the language in ways that make queries tractable. This is what the authors set out to do in the submitted paper.