Convex Polytope Trees

1 Introduction . Decision trees [ 8 ] are highly interpretable models , which make them favorable in high-stakes domains such as medicine [ 32 , 43 ] and criminal justice [ 6 ] . They are also resistant , if not completely immune , to the inclusion of many irrelevant predictor variables . However , trees usually do not have high accuracy , which somewhat limits their use in practice . Current main approaches to improve the performance of decision trees are making large trees or using ensemble methods [ 11 , 12 , 18 , 53 ] , such as bagging [ 7 ] and boosting [ 13 , 16 ] , which come with the price of harming model interpretability . There is a trade-off challenge between the accuracy and interpretability of a decision tree . Prior work has attempted to address the aforementioned challenge and improve the performance of trees by introducing oblique tree models [ 19 , 34 ] . These families of models are generalizations of classical trees , where the decision boundaries are hyperplanes that are not constrained to be axis-parallel and can have an arbitrary orientation . This change in the decision boundaries has been shown to reduce the size of trees . However , the tree size often remains too large in real datasets to make it amenable to interpretation . There has been an extensive body of research to improve the training of the oblique trees and enhance their performance [ 5 , 9 , 28 , 46 ] , yet their large size remains a challenge . In this paper , we propose convex polytope decision trees ( CPT ) to expand the class of oblique trees by extending hyperplane cuts to more flexible geometric shapes . To be more specific , the decision 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . boundaries induced by each internal node of CPT are based on noisy-OR [ 36 ] of multiple linear classifiers . And since noisy-OR has been widely accepted as an interpretable Bayesian model [ 37 ] , our generalization keeps the interpretability of oblique trees intact . Furthermore , CPT ’ s decision boundaries geometrically resemble a convex polytope ( i.e. , high dimensional convex polygon ) . Therefore , the decisions at each node have both logical and geometrical interpretation . We use the gamma process [ 14 , 25 , 52 , 51 ] , a nonparametric Bayesian prior , to infer the number of polytope facets adaptively at each internal tree node and regularize the capacity of the proposed CPT . A realization of the gamma process consists of countably infinite atoms , each of which is used to represent a weighted hyperplane of a convex polytope . The shrinkage property of the gamma process helps us to encourage having simpler decision boundaries , therefore help resist overfitting and improve interpretability . Figure 1 provides an illustrative example of our model for a toy data . Note , our main goal for proposing CPT is not to improve interpretability by having a smaller number of leaves rather it is pushing accuracy limits of decsion trees methods while staying in the class of interpretable models . Neural networks , with more than two layers , are often considered black-box models ( not in the class of interpretable models ) . However , even with larger depth , lots of leaves , or complex ( but interpretable ) decision boundaries , decision trees would still remain in the class of interpretable models . Our proposed CPT offers higher accuracy than previously studied tree models and remains interpretable ( noisy-OR decision boundaries are widely considered interpretable ) . While it is hard to objectively compare the interpretability ( “ Interpretability is a domain-specific notion so there can not be an all-purpose definition ” [ 38 ] ) of convex polytope trees vs. oblique trees , they offer an alternative where the ease of interpretability of the decision boundaries at each node can be traded for shallower trees with significantly fewer leaves . Choosing one over the other is often application dependent , with the added advantage that convex polytope trees often offer higher accuracy . The training of CPT , like that of oblique trees , is a challenging task because it requires learning both the structure ( i.e. , the topology of the tree and the cut-off for the decision boundaries ) and the parameters ( i.e. , parameters of noisy-OR ) . The structure is a discrete optimization problem , involving the search over a potentially large problem space . In this work , we present two fully differentiable approaches for learning CPT models , one based on mutual information maximization , applicable for both binary and multi-class classification , and the other based on variance minimization , applicable for regression . The differentiable training allows one to use modern stochastic gradient descent ( SGD ) based programming frameworks and optimization methods for learning the proposed decision trees for both classification and regression . Experimentally , we compare the performance of CPT to state-of-the-art decision tree algorithms [ 9 , 27 ] on a variety of representative tasks in both regression and classification . We experiment with several real-world datasets from diverse domains , such as computer vision , tabular data , and chemical property data . Experiments demonstrate that CPT outperforms state-of-the-art methods with higher accuracy and smaller size . Our main contributions include : 1 ) We propose an interpretable generalization to the family of oblique decision trees models ; 2 ) We regularize the expressive power of CPT , using a nonparametric Bayesian shrinkage prior for each node split function ; 3 ) We provide two scalable and differentiable ways of learning CPT models , one for classification and the other for regression , which efficiently search for the optimal tree ; 4 ) We experimentally evaluate CPT on several different types of predictive tasks , illustrating that this new approach outperforms the prior work in having higher accuracy achieved with a smaller size . 2 Related work . Most decision tree literature focuses on training a single or an ensemble of trees [ 18 ] instead of making decision boundaries more flexible . One reason for this lack of research is the increase in the problem ’ s computational complexity , even for simple generalizations of the decision boundary . For example , with N as the number of data points and d as the dimension of the input space , generalizing the coordinate-wise to an oblique hyperplane cut , increases the number of possible splits of data points from Nd to Pd i=0 N i just for a single node [ 44 ] . To have more flexible decision boundaries , some methods perform hyperplane cuts in an extended feature space , created by concatenating the original and newly generated features [ 1 ] . These new features are engineered or kernel-based and not designed for interpretability . Therefore unlike CPT , they are not an interpretable extension of the decision tree . We follow this section with a literature review of the recent training algorithms for axis-aligned and oblique trees . Conventional methods for decision tree induction are greedy , where they grow the tree nodes one at a time . The greedy construction of oblique trees can be done by using coordinate descent to learn the parameters of each split [ 34 ] , or by a projection of the feature space to a lower dimension then using coordinate-cut [ 33 , 46 ] . However , the greedy procedure often leads to sub-optimal trees . There have been several attempts to non-greedy optimization , which rely on either fuzzy or probabilistic split functions [ 22 , 26 , 40 , 41 ] . The probabilistic trees are sometimes referred to as soft decision trees [ 17 , 20 , 26 ] . In these methods , the assignment of a single sample to the leaf is fuzzy or probabilistic , and gradient descent is used to optimize the tree . Most of these algorithms remain probabilistic at test time , leading to uninterpretable models as the prediction for each sample will be based on multiple leaves of the tree instead of just one . Other advances towards the training of an oblique tree are based on constructing neural networks that reproduce decision trees [ 28 , 48 ] . Yang et al . [ 48 ] use a neural network with argmax activations for the representation of classic decision tree with coordinate cuts , but they are not scalable to highdimensional data . Lee and Jaakkola [ 28 ] use the gradient of a ReLU network with a single hidden unit at each layer and skip-connections to construct an oblique decision tree . They achieve state-of-the-art results on some datasets but have to make complete trees with a high depth , consequently having a large number of leaves . In contrast to our method , there are other training algorithms that require the tree ’ s structure at the beginning . Some of the works in this direction like Bennett [ 4 ] and Bertsimas and Dunn [ 5 ] use linear programming , or mixed-integer linear programming , to find an optimum global tree . Therefore , these methods are computationally expensive and not scalable . Norouzi et al . [ 35 ] derive a convex-concave upper bound on the tree ’ s empirical loss and optimize that loss using SGD . A recent work [ 9 ] proposes tree alternating optimization , where one directly optimizes the misclassification error over separable subsets of nodes , achieving the state-of-the-art empirical performance on several datasets in a comprehensive comparison [ 49 ] . We conclude this section by connecting our splitting rule at each internal node to relevant classification algorithms [ 2 , 23 , 30 , 31 , 45 , 51 ] . The two most related works are convex polytope machine ( CPM ) [ 23 ] and infinite support hyperplane machine ( iSHM ) [ 51 ] , which both exploit the idea of learning a convex polytope associated decision boundary . iSHM is like a single hidden layer NN and is restricted to the binary classification task . 3 Convex Polytope Tree and Its Inference Algorithms . Suppose we are given the training data ( X , Y ) = { ( xn , yn ) } Nn=1 , where pairs of ( xn , yn ) are drawn independently from an identical and unknown distribution D. Each xn 2 Rd is a d-dimensional data with a corresponding label yn 2 Y . In the classification setting , Y = { 1 , · · · , K } and in regression scenario Y = R. The aim is to learn a function F : Rd ! Y that will perform well in predicting the label on samples from D. Decision tree methods construct the function F by recursively partitioning the feature space to yield a number of hierarchical , disjoint regions and assign a single label ( value ) to each region . In what follows , we will explain how we move beyond the oblique decision trees . 3.1 Convex Polytope Constrained Decision Boundary . By extending the idea of disjunctive interaction ( noisy-OR , probabilistic-OR ) [ 39 , 21 , 50 , 51 ] from probabilistic reasoning to the decision tree problem , we make the decision boundaries more flexible while preserving interpretability . To that end , we transform the problem of splitting at each node ( right or left ) to a committee of experts that make individual binary decisions ( “ Yes ” or “ No ” ) . The committee votes “ Yes ” if and only if at least one expert votes “ Yes , ” otherwise it votes “ No. ” Thus , the final vote at each node is vote = _K i=1 votei , where W denotes the logical OR operator . We model each expert as a linear classifier who votes “ Yes ” with probability P ( votei = “ Yes ” | { ri , i } , x ) = 1 ( 1 + e 0 ix ) ri , ( 1 ) where ri 0 and i 2 Rd are parameters of expert i . Now assuming that each expert votes independently , we can express the probability of the committee voting “ Yes ” as P ( vote = “ Yes ” | { ri , i } i , x ) = 1 QK i=1 ( 1 pi ) = 1 e PK i=1 ri ln ( 1+e 0ix ) , ( 2 ) where pi is the probability of expert i voting “ Yes ” and K is the total number of experts . We can now define the split function at each node by thresholding the committee voting probability : Aleft : = { x |x 2 Rd , P ( vote = “ Yes ” | { ri , i } i , x ) qthr } ( 3 ) where Aleft and Aright : = Rd \Aleft are the related splits of the space . To elaborate on the geometric shape and interpretability of the decision boundaries , consider K = 1 ( i.e. , a single expert ) . In this scenario , the decision boundary becomes a hyperplane , which is perpendicular to . In fact , the probability function for each expert is based on the signed distance of x to the hyperplane perpendicular to . Moreover , the parameter r controls how smoothly the probability transitions from 0 to 1 , where a larger r leads to sharper changes . This class of models with K = 1 and r = 1 are identical to oblique trees , which are interpretable . Furthermore , when K 1 , the interpretability is provided by the fact that linear classifiers and probabilistic-OR operation are both interpretable [ 37 ] . To geometrically analyze the implied decision regions , we provide the following theorem . Theorem 1 . For any { ri , i } Ki=1 , such that ri 2 R+ and i 2 Rd , let : Aleft = { x |x 2 Rd , f ( x ) qthr } , where : f ( x ) = 1 e PK i=1 ri ln ( 1+e 0ix ) ( 4 ) then Aleft is a convex set , confined by a convex polytope . The proof is provided in the Appendix . The above theorem shows that for K 1 , the decision region ( Aleft ) is a convex set confined by a convex K-sided polytope . More precisely , each facet of the convex polytope is a hyperplane corresponding to an expert perpendicular to its . Also worth noting , an expert with a larger r has more effect on the decision boundary , making sharper changes to the probability function . This can also be perceived as the value of their decision in the committee . Therefore , our method not only has a strong relationship with probabilistic-OR type models that provide interpretability for the model parameters [ 3 ] but also has decision boundaries with interpretable geometric characteristics . We propose a class of models , Convex Polytope Trees ( CPT ) , where each tree node follows the above splitting function .

Summary: The authors propose an alternative decision tree model, where the splitting criteria/decision boundary are convex polytopes instead of axis aligned halfplanes or even more general oblique hyperplanes.  One advantage of this is that it gives us another mechanism to add model capacity (for traditional decision trees, model capacity is typically controlled by adding trees or increasing depth). The proposed approach adds capacity by making the splitting criterion/decision boundary more flexible.  As they show via experiments, this leads to requiring fewer trees/leaf nodes, which could potentially lead to more interpretable models.  In addition, their technique can use SGD based techniques allowing it to be used in conjunction with other (differentiable) downstream techniques, not to mention potentially being used for larger datasets.