Discovering Classification Rules for Interpretable Learning with Linear Programming

1 INTRODUCTION . Medical diagnosis , educational and juridical decisions often have important consequences for the society . Therefore , both the accuracy and interpretability of these decisions are of crucial importance . In particular , these decisions should be understandable by the decision makers . Rule sets consisting of a few intuitively coherent rules have shown to accomplish this purpose in such domains ( Lakkaraju et al. , 2016 ) . Here , we first aim at rule extraction from powerful ensemble methods and then solely focus on rule generation . In both cases , our objective is to obtain a set of classification rules that balances the trade-off between accuracy and interpretability . Our main tools in this effort are linear programming and column generation . A rule is an independent if-then statement , which contains one or more conditions that assign a class to a set of samples . For example in binary classification , “ if ( Clump Thickness is greater than six ) and ( Single Epithelial Cell Size is less than four ) then the tumor is malignant ” is such a rule that can be used for breast cancer diagnosis . When a sample satisfies this rule , then it receives a label corresponding to one of the two classes . In case a sample is covered by more than one rule , then majority voting among the assigned labels is used to determine the class of the sample . Growing decision trees is closely related to rule-based learning . A Decision Tree ( DT ) naturally results with a set of leaves , where each leaf corresponds to a different rule . On one hand , rule learning is considered to be more flexible for interpretability than tree-based learning approaches . Fürnkranz ( 1999 ) lists superiorities of rule-based learning over tree-based learning . Leaves ( rules ) of a DT are not independent from each other . For example , in a binary tree when a splitting is performed at a node , the left child grows a rule , while its right sibling grows its negation . As a result , every sample obeys to exactly one rule and it is classified according to the corresponding leaf . Thus , inaccurate rules , which are negations of their siblings after splitting , can be created . This may render false classification of samples and reduce both accuracy and interpretability . However , independently constructed rules do not need such a negation rule , and hence , rule-based learning can be considered more flexible for interpretability . On the other hand , unlike DTs , an independent set of rules does not necessarily cover the entire sample space . This may result in a state where a test sample can not be classified with the proposed rule set . This drawback is often handled by assigning a default label to those uncovered samples ( Fürnkranz et al. , 2012 ) . To minimize the number of uncovered samples during testing , the training data is required to be fully covered . Separate-andconquer algorithm by Fürnkranz ( 1999 ) achieves this heuristically by fitting rules on uncovered training samples , and new rules are generated until each training sample is covered by at least one rule . Instead of such a sequential covering , we take advantage of our linear programming ( LP ) approach and explicitly impose a covering constraint on the training set . In this paper , we propose two algorithms for interpretability and learning that are based on mathematical programming . We give a generic LP model that is first used by our Rule Extraction ( RUX ) algorithm , which selects rules from trained tree or rule ensembles for interpretation . Then , we develop a Rule Generation ( RUG ) algorithm , which generates classification rules using column generation ( CG ) to solve the LP model . Rule extraction methods attempt to select a set of rules from accurate complex or black-box models to interpret the predictions of these models ( Hayashi & Oishi , 2018 ) . For instance , several works in the literature aim at interpreting Random Forest ( RF ) models by extracting rules from the trees in the forest ( Liu et al. , 2012 ; Lu Thi et al. , 2015 ; Adnan & Islam , 2017 ; Wang et al. , 2020 ; Birbil et al. , 2020 ) . Most of these studies use heuristic approaches to select the desired set of rules . Birbil et al . ( 2020 ) suggest a set covering formulation to extract intrepretable rules from RFs . Other applications include extraction of rules from artificial neural networks ( Andrews et al. , 1995 ) and support vector machines ( Barakat & Bradley , 2010 ) . As a remark , there are other post-hoc interpretability approaches in the literature such as SHAP ( Lundberg & Lee , 2017 ) and LIME ( Ribeiro et al. , 2016 ) that are considered to be model-agnostic . However , our RUX algorithm is a model-specific approach . There are several studies that are closely related to ours , since they also employ mathematical programming for rule learning . Malioutov & Varshney ( 2013 ) propose a rule-based binary classifier solving a binary program that minimizes the number of rules for a boolean compressed sensing problem . Wang & Rudin ( 2015 ) present a mixed integer linear programming ( MILP ) formulation to learn decision rules for binary classification ( e.g. , patterns ) . They give a discussion on how their classifier is equivalent to DTs and RFs . The MILP formulation is solved using generated rules with an objective of minimizing the number of misclassified samples , the number of rules generated , and the total length of each rule . Dash et al . ( 2020 ) offer a CG-based framework to find optimal rule set for binary classification , where the objective is to find a trade-off between classification rule simplicity and accuracy . One-hot encoding is used to binarize categorical data , and numerical data is also discretized with sample deciles as thresholds . For large instances , the pricing subproblem is either solved with time limits , or the model columns are generated by a greedy heuristic . Wei et al . ( 2019 ) propose generalized linear rule models using a similar CG framework of the work by Dash et al . ( 2020 ) . Malioutov & Meel ( 2018 ) solve a Max-Sat formulation by constraint programming to construct interpretable classification rules . Ghosh & Meel ( 2019 ) also propose a framework based on MaxSAT formulation that can be applied to binary classification problems with binary features to minimize the number of generated rules and the number of misclassified samples . Their approach is incremental and takes advantage of partitioning the data set into several clusters . Contributions . RUX and RUG are based on an LP model and thus , both algorithms are scalable for large datasets . This is an important advantage compared to the existing studies that use MILP formulations . The proposed algorithms directly address multi-class problems while the existing studies using optimization-based approaches are for binary classification . To that end , the objective function in our LP formulation minimizes classification error using a loss function instead of explicitly counting misclassified samples . Both RUX and RUG can work with continuous or categorical features , and hence , they do not require encoding of the data . Along with the set of rules , our algorithms also return the optimal weights for the rules . These weights allow to attach importance to each rule for interpreting the classification . Our algorithms admit assigning cost coefficients to the rules . These coefficients could relate to different attributes of the rules , such as ; rule lengths , estimator weights , number of false negatives , and so on . The objective function also allows penaliz- ing rules that may have undesired outcomes ( like long rule lengths ) . Thus , the decision makers can use these coefficients to lead the training process to obtain a set of rules that are more appealing for their needs . The novelty in our column generation approach is the use of a regular decision tree with sample weights as the pricing subproblem . Training trees with sample weights is very fast , and also standard in all machine learning packages as boosting methods also rely on sample weights . Lastly , we present our algorithms for multi-class classification problems . However , the proposed ideas can also be extended to discovering regression rules for interpretable learning with linear programming . 2 RULE EXTRACTION . We consider a classification problem with K classes and denote the set of class labels by K. The training dataset consists of samples with features xi ∈ Rp for i ∈ I and labels yi for i ∈ I . To work with multiple classes , we define a vector-valued mapping y ( xi ) ∈ K ⊂ RK as in Zhu et al . ( 2009 ) . That is , if yi = k , then y ( xi ) = ( − 1K−1 , − 1 K−1 , . . . , 1 , . . . , − 1 K−1 ) ᵀ , ( 1 ) where the value one appears only at the kth component of the vector . Suppose that we have a collection of rules indexed by J . A rule j ∈ J assigns the vectorRj ( xi ) ∈ K to input xi , only if rule j covers sample i . This vector is also formed in the same manner as in equation 1 . To predict the class of a given sample xi with this collection of rules , we use a set of nonnegative weights wj , j ∈ J associated with the rules and evaluate ŷ ( xi ) = ∑ j∈J aijRj ( xi ) wj , ( 2 ) where aij ∈ { 0 , 1 } indicates whether rule j covers sample i or not . Then , the index of the largest component of the resulting vector ŷ ( xi ) is assigned as the predicted label ŷi of sample i ∈ I . Note that equation 2 is similar to the weighting of the classifiers in standard boosting methods . Here , instead of classifiers , we use rules for classifying only the covered samples . In order to evaluate the classification error , we use the hinge loss and define the total classification loss by ∑ i∈I max { 1− ∑ j∈J âijwj , 0 } , where âij = κaijRj ( xi ) ᵀy ( xi ) with κ = ( K−1 ) /K . This loss function allows us to write a linear programming model , where the objective is to find the set of rules that minimizes the total loss . To this end , we introduce the auxiliary variables vi , i ∈ I standing for vi ≥ max { 1− ∑ j∈J âijwj , 0 } , and obtain our master problem minimize ∑ i∈I vi + ∑ j∈J cjwj subject to ∑ j∈J âijwj + vi ≥ 1 , i ∈ I ; ∑ j∈J aijwj ≥ ε , i ∈ I ; vi ≥ 0 , i ∈ I ; wj ≥ 0 , j ∈ J , ( 3 ) where cj ≥ 0 , j ∈ J are the cost coefficients . These coefficients and the second set of constraints require further explanation . The cost coefficients serve two important roles : First , solutions involving many rules with nonzero weights are avoided . The less the number of rules in the resulting set , the easier it is to interpret a solution . In other words , we prefer sparse solutions and cj serves to that preference . Second , in many different application domains , rules have actual costs that need to be taken into consideration . As we also highlight in our title , when interpretability is of concern , one could try to obtain rules with few features because shorter rules are considered to be easier to interpret ( Lakkaraju et al. , 2016 ) . In this case , the number of conditions in a rule can be set as the rule cost . As another example , consider a classification problem in medical diagnosis , where false negatives are likely to be perceived as more costly than false positives . Such an evaluation could also be easily incorporated with a rule cost in the proposed model . We should point out that there exists a trade-off between model accuracy and the rule set size . This can also be handled with our formulation by introducing a fixed cost coefficient , i.e. , using the same value for all cj , j ∈ J . The master problem equation 3 also involves a set of covering constraints with the fixed right-handside , ε > 0 . These constraints make sure that each sample is covered by at least one rule . The need for these covering constraints is exemplified as follows : Consider a binary classification problem , where we select three samples { xi , xk , xl } along with their labels yi = 1 , yk = −1 and yl = −1 . Suppose that we have two rules j and j′ such that the former covers all three , whereas the latter covers only the last two samples . Here , the rules use majority voting as it is applied to the leaves in trained DTs . The labels assigned by the rules j and j′ are as follows : Rj ( xi ) = Rj ( xk ) = Rj ( xl ) = −1 and Rj′ ( xk ) = Rj′ ( xl ) = −1 . The first set of constraints in equation 3 then becomes −aijwj +vi ≥ 1 , akjwj +akj′wj′ +vk ≥ 1 , aljwj +alj′wj′ +vl ≥ 1 . For simplicity , if we also assume that cj = cj′ , then the optimal solution becomes wj = vk = vl = 0 and wj′ = vi = 1 . With this solution , sample xi is not covered . We remark that covering each sample in a dataset is crucial from the perspective of giving a literal interpretation with the obtained rules . This is particularly important when the resulting set of rules is used to interpret the classification of individual samples in the dataset . Clearly , varying ε may lead to a change in the set of optimal rules . For instance a large ε value may impose larger rule weights on those samples covered with only few rules . However , we point out that the role of this constraint is just coverage , and hence , setting ε to a small strictly positive value is sufficient1 . Up until this point , we have not specified any details about the rule set J . On one hand , this rule set can be static in the sense that it can be obtained by extracting the rules available through a trained tree ensemble algorithm or by using the rules resulting from a rule-based method ( Cohen & Singer , 1999 ) . For instance , consider a RF model trained on a given dataset . Then , the rule set J can be obtained from the leaves of the trees in the forest , since each leaf corresponds to a rule . Solving equation 3 with such a rule set allows us to extract the rules that were most critical for classification . As we have mentioned before , we can assign the length of a rule as its cost and try to obtain rules with desirable lengths for interpretation . In a similar vein , consider another example with a trained AdaBoost ( Freund & Schapire , 1997 ) model for which the base estimators are set as DTs . Again , the leaves of the trees from AdaBoost can be used to construct the rule set J in equation 3 , which is then solved to extract an interpretable set of rules . The costs of the rules in this case could be the inverse of the estimator weights assigned by the AdaBoost algorithm to the trees . In this way , the obtained set of rules is more likely to inherit the importance of the DTs from the AdaBoost model . Using these trained models to construct our master problem leads to our first algorithm RUX , which is based on tree ensembles . We show in Section 4 that RUX can indeed extract only a selection of rules from the trained models without significantly sacrificing accuracy . The computational complexity for RUX is determined by the underlying LP solver used . In practice , most of the LP solvers use interior point methods that work in polynomial-time .

Summary: The paper presents an approach to learning ensembles of classification rules using linear programming. Two variants of the algorithm are considered: one uses a collection of rules extracted from ensembles of decision trees as the starting point, and another one that learns rules from scratch by applying decision tree learning within the rule learning algorithm. UCI datasets (mostly two-class problems) are used to compare classifier complexity and accuracy with that of random forests, AdaBoost, and learning a single decision tree.