Learning Interpretable Decision Rule Sets: A Submodular Optimization Approach

1 Introduction . Interpretability is becoming one of the key considerations when deploying machine learning models to high-stake decision-making scenarios . Black box models , such as random forests and deep neural networks , may achieve impressive prediction accuracy in practice , but it is often difficult to understand the mechanisms about how their predictions are made . Moreover , it has been widely known that machine learning models are susceptible to spurious correlations , which makes them not robust to distribution shifts or adversarial attacks and leads to misleading predictions . Black box models are particularly problematic here because they are hard to audit and to diagnose . The development of inherently interpretable models is a longstanding attempt towards interpretable and trustable machine learning [ 42 , 43 ] . In this paper , we consider decision rule sets for binary classification , in which if an example is tested true for a Boolean condition expressed in disjunctive normal form ( DNF ) , then a predefined label is predicted for it . For instance , a rule set for determining whether a patient with community-acquired pneumonia should be hospitalized may be `` IF ( IsChild = True AND OxygenSaturation < 0.9 ) OR ( IsChild = True AND SOB = True ) OR ( IsAdult = True AND CURB65 > 2 ) THEN Yes '' . Rule sets are particularly suited for tabular data that contain mixed-type features and exhibit complex high-order feature interactions . When compared to other rule-based models such as decision trees and decision lists , the simpler combinatorial structure of rule sets makes them easier to interpret and to learn from data . The learning of rule sets has attracted interest from various research communities over the past few decades . Two interplaying subset selection tasks constitute the major challenges in rule set learning . First , the construction of a rule requires choosing a subset of all features . Then a subset of all possible 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . rules has to be selected to form a rule set . In early algorithms from machine learning community , such as FOIL [ 41 ] , CN2 [ 12 ] and RIPPER [ 13 ] , usually a greedy sequential covering strategy is utilized , in which at each stage a rule is generated from uncovered examples using heuristics and newly covered examples are removed . Associative classification techniques [ 33 , 50 ] developed by data mining community take a different two-stage strategy , in which a large set of rules are first generated via association rule mining and then a rule set is constructed by ranking and pruning . Both these early approaches lack a global objective that guides the generation of rules and optimizes the interpretability of produced rule set . This paper presents a new submodular optimization perspective on nearly optimal rule set learning . We employ an objective function that simultaneously optimizes accuracy and interpretability . We show that given a ground set of rules , this objective is a regularized submodular function [ 27 ] , for which algorithms with strong approximation guarantees have been developed recently [ 23 ] . Instead of using a pre-mined pool of rules as the ground set , we take an on-the-fly approach in which the ground set consists of all possible rules and elements are picked from it one at a time through solving an optimization subproblem . When the subproblem is always solved to optimality , our learning algorithm enjoys the algorithmic guarantees for regularized submodular maximization . For most real cases the global optimum of the subproblem is impractical to reach , therefore we propose an efficient approximate algorithm based on iterative local combinatorial search . Our contributions are summarized as follows : ( i ) We formulate interpretable rule set learning as a regularized submodular maximization problem which is amenable to approximate algorithm with strong guarantees . ( ii ) We discover an intuitive difference of submodular ( DS ) decomposition for the induced rule construction subobjective , based on which an iterative refinement algorithm is proposed to solve the subproblem approximately for large datasets . ( iii ) We conduct a comprehensive experimental evaluation , demonstrating the proposed approach is competitive against state-of-the-arts in both predictive performance and interpretability . The remainder of this paper is organized as follows : Related work is summarized in Section 2 . Section 3 introduces the problem formulation . Section 4 presents the algorithmic details . Experimental evaluation is reported in Section 5 . We discuss the limitations of this work in Section 6 and conclude the paper with Section 7 . 2 Related work . To discover rules from training data , heuristic sequential covering or separate-and-conquer strategy is utilized by almost all early algorithms [ 41 , 12 , 13 ] , in which the interpretability of final rule set is not explicitly controlled . In recent years , with the renewed interest in interpretable rule-based models , the rule set learning problem was revisited under modern optimization lens and several new algorithms were developed [ 22 , 49 , 28 , 35 , 48 , 51 ] . These algorithms mainly follow a two-stage paradigm originating from early associative classification methods [ 33 , 50 ] , in which the learning problem is divided into two separate steps : rule generation and rule selection . For example , Lakkaraju et al . [ 28 ] frame the task of constructing a interpretable rule set from pre-mined rules as a submodular optimization problem and solve it approximately with a local search algorithm . In general , such two-stage paradigm suffers from the drawback that some essential rules may be filtered out prematurely . Dash et al . [ 14 ] formulate the rule set learning problem as an integer program ( IP ) and solve its linear programming relaxation using the column generation ( CG ) framework [ 15 , 17 ] , in which rules are instead generated on the fly by iteratively solving a IP subproblem . In this way they bridge rule generation and rule selection together as a unified optimization procedure and thus avoid the drawbacks of pre-mining . We contribute a similar framework from a different submodular optimization perspective . Interestingly , subproblems in both frameworks share the same form ( 6 ) , which asks the solution to cover more positively weighted samples and less negatively weighted samples . This task shares some similarities with the subgroup discovery [ 6 ] problem in data mining , for which exhaustive search [ 7 , 31 ] , heuristic beam search [ 30 , 46 ] and sampling [ 9 , 10 ] are commonly utilized . Eckstein and Goldberg [ 17 ] designed a specialized branch-and-bound method for the closely related maximum monomial agreement ( MMA ) problem . To the best of our knowledge , we are the first to study this task from a DS optimization perspective . Our work is also related to optimization based learning of other rule models , such as optimal decision lists [ 3 , 4 ] and optimal decision trees [ 24 , 34 , 8 , 39 , 2 , 47 , 52 ] . The learning of decision lists and trees is inherently much harder because their combinatorial structures are more complex than rule sets . 3 Problem formulation . In this work , we consider binary classification based on a set of interpretable binary features . Categorical features can be easily binarized via one-hot encoding , and numerical features may be binarized through bucketing or comparisons like `` Age ≥ 18 '' . Techniques such as one-versus-one are available for transforming multiclass classification to binary classification . Suppose we are given the training data { ( xi , yi ) } ni=1 , in which each xi = ( xi,1 , . . . , xi , d ) ∈ { 0 , 1 } d is a binary feature vector associated with a label yi ∈ { 0 , 1 } . The goal is to learn an interpretable rule set model F : { 0 , 1 } d → { 0 , 1 } to predict the label . A rule set is a Boolean classifier of the form `` IF P ( x ) THEN y = 1 ELSE y = 0 '' , in which P ( x ) is a predicate restricted in disjunctive normal form ( DNF ) and treats raw features { xj } dj=1 as its variables . For simplicity , the use of not ( ¬ ) operator in DNF is disabled because equivalent expressivity can be achieved by creating augmented features { ¬xj } dj=1 . Then a concrete P ( x ) is described as a set S ⊆ 2 [ d ] , where each element R ∈ S is a subset of [ d ] : = { 1 , . . . , d } . The mapping from a S to a DNF is given by PS ( x ) : = ∨ R∈S ∧ j∈R xj . We call S a rule set and call each element R of it a rule . A sample x is covered by a rule R if P { R } ( x ) is true . It is now clear that a rule set classifier can be simplified to y = PS ( x ) , i.e. , it predicts 1 iff x is covered by at least one of the rules . To learn a rule set from data , we consider minimizing the empirical error while penalizing the complexity by solving the optimization problem : min S⊆2 [ d ] n∑ i=1 l ( PS ( xi ) , yi ) + Ω ( S ) ( 1 ) However , even under the simplest setting where l ( · , · ) is the 0-1 loss and Ω ( · ) is the cardinality function , this problem is still hard to work with . To proceed , we replace PS with a surrogate model P̃S : { 0 , 1 } d → N0 defined as P̃S ( x ) = ∑ R∈S ∧ j∈R xj . Then a flexible objective function that balances accuracy and interpretability is chosen to be : L ( S ) = n∑ i=1 lβ ( P̃S ( xi ) , yi ) + λ ∑ R∈S |R| ( 2 ) where lβ ( · , · ) is a loss function with hyperparameters β = ( β0 , β1 , β2 ) ∈ R3+ : lβ ( ŷ , y ) = β0ŷ if y = 0 β1 ( y − ŷ ) if y = 1 ∧ ŷ ≤ 1 β2 ( ŷ − y ) if y = 1 ∧ ŷ > 1 With a proper choice of hyperparameters , interpretability of the minimizer for this objective comes from two aspects : sparsity and diversity . Sparsity is obvious because of the complexity regularization term . Diversity is encouraged by the loss function through penalizing the overlap of rules . P̃S ( x ) > 1 means that x is covered by more than one of the rules , and in this case the objective function will receive a penalty . Small overlap among rules benefits interpretability because it makes the decision boundaries more clear and enables people to inspect the learned rules individually . It may not be immediately clear thatL ( S ) is a supermodular function and thus−L ( S ) is a submodular function . To show this , we introduce more notations . Let X+ = { i|yi = 1 } and X− = { i|yi = 0 } be the set of positive and negative samples , respectively . Let X+S = { i|i ∈ X+ ∧ PS ( xi ) } and X−S = { i|i ∈ X− ∧ PS ( xi ) } be the set of positive and negative samples covered by S , respectively . Then we have L ( S ) = β0 ∑ R∈S ∣∣∣X− { R } ∣∣∣+ β1 ∣∣X+ \ X+S ∣∣+ β2 ( ∑ R∈S ∣∣∣X+ { R } ∣∣∣− ∣∣X+S ∣∣ ) + λ ∑ R∈S |R| Ignoring the weights , then the first term is the miscoverage of negative samples , the second term is the number of uncovered positive samples , the third term is the overlap in positive samples , and the last term is the size of rules . After reorganizing the terms , we arrive at L ( S ) = β1 ∣∣X+∣∣− ( β1 + β2 ) ∣∣X+S ∣∣+ ∑ R∈S β0 ∣∣∣X− { R } ∣∣∣+ β2 ∣∣∣X+ { R } ∣∣∣+ λ|R| ( 3 ) Then the submodularity immediately follows . Proposition 1 . L ( S ) is a supermodular function and −L ( S ) is a submodular function . Proof . The first term in ( 3 ) is a constant . The second term is a negative coverage , while coverage functions are well-known to be submodular . The third term sums over elements of S and thus is modular . A submodular function minus a modular function is still submodular , therefore −L ( S ) is submodular and L ( S ) is supermodular . We show in the Appendix that several design choices for the original objective in ( 1 ) may reduce to L ( S ) through upper bounding surrogates , which justifies the generality of this objective function . This objective function also generalizes the Hamming loss used in previous work [ 44 , 14 ] .

Summary: This paper addresses learning rule set models for binary classification, ie combinations of logical rules on features to classify a positive class. Since learning rule sets requires selection of features over which the logical rules are applied and selection of the rules themselves, the subset selection problem is exponential. This paper formulates the task loss as a supermodular function comprised of a constant term (ignored in the optimization), a modular function term and a submodular function term. The paper outlines an optimization algorithm that contains a subproblem that is not computationally feasible except for small problems. Thus, it outlines an approximate solution to the subproblem.