Convex Polytope Trees

1 Introduction . Decision trees [ 8 ] are highly interpretable models , which make them favorable in high-stakes domains such as medicine [ 32 , 43 ] and criminal justice [ 6 ] . They are also resistant , if not completely immune , to the inclusion of many irrelevant predictor variables . However , trees usually do not have high accuracy , which somewhat limits their use in practice . Current main approaches to improve the performance of decision trees are making large trees or using ensemble methods [ 11 , 12 , 18 , 53 ] , such as bagging [ 7 ] and boosting [ 13 , 16 ] , which come with the price of harming model interpretability . There is a trade-off challenge between the accuracy and interpretability of a decision tree . Prior work has attempted to address the aforementioned challenge and improve the performance of trees by introducing oblique tree models [ 19 , 34 ] . These families of models are generalizations of classical trees , where the decision boundaries are hyperplanes that are not constrained to be axis-parallel and can have an arbitrary orientation . This change in the decision boundaries has been shown to reduce the size of trees . However , the tree size often remains too large in real datasets to make it amenable to interpretation . There has been an extensive body of research to improve the training of the oblique trees and enhance their performance [ 5 , 9 , 28 , 46 ] , yet their large size remains a challenge . In this paper , we propose convex polytope decision trees ( CPT ) to expand the class of oblique trees by extending hyperplane cuts to more flexible geometric shapes . To be more specific , the decision 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . boundaries induced by each internal node of CPT are based on noisy-OR [ 36 ] of multiple linear classifiers . And since noisy-OR has been widely accepted as an interpretable Bayesian model [ 37 ] , our generalization keeps the interpretability of oblique trees intact . Furthermore , CPT ’ s decision boundaries geometrically resemble a convex polytope ( i.e. , high dimensional convex polygon ) . Therefore , the decisions at each node have both logical and geometrical interpretation . We use the gamma process [ 14 , 25 , 52 , 51 ] , a nonparametric Bayesian prior , to infer the number of polytope facets adaptively at each internal tree node and regularize the capacity of the proposed CPT . A realization of the gamma process consists of countably infinite atoms , each of which is used to represent a weighted hyperplane of a convex polytope . The shrinkage property of the gamma process helps us to encourage having simpler decision boundaries , therefore help resist overfitting and improve interpretability . Figure 1 provides an illustrative example of our model for a toy data . Note , our main goal for proposing CPT is not to improve interpretability by having a smaller number of leaves rather it is pushing accuracy limits of decsion trees methods while staying in the class of interpretable models . Neural networks , with more than two layers , are often considered black-box models ( not in the class of interpretable models ) . However , even with larger depth , lots of leaves , or complex ( but interpretable ) decision boundaries , decision trees would still remain in the class of interpretable models . Our proposed CPT offers higher accuracy than previously studied tree models and remains interpretable ( noisy-OR decision boundaries are widely considered interpretable ) . While it is hard to objectively compare the interpretability ( “ Interpretability is a domain-specific notion so there can not be an all-purpose definition ” [ 38 ] ) of convex polytope trees vs. oblique trees , they offer an alternative where the ease of interpretability of the decision boundaries at each node can be traded for shallower trees with significantly fewer leaves . Choosing one over the other is often application dependent , with the added advantage that convex polytope trees often offer higher accuracy . The training of CPT , like that of oblique trees , is a challenging task because it requires learning both the structure ( i.e. , the topology of the tree and the cut-off for the decision boundaries ) and the parameters ( i.e. , parameters of noisy-OR ) . The structure is a discrete optimization problem , involving the search over a potentially large problem space . In this work , we present two fully differentiable approaches for learning CPT models , one based on mutual information maximization , applicable for both binary and multi-class classification , and the other based on variance minimization , applicable for regression . The differentiable training allows one to use modern stochastic gradient descent ( SGD ) based programming frameworks and optimization methods for learning the proposed decision trees for both classification and regression . Experimentally , we compare the performance of CPT to state-of-the-art decision tree algorithms [ 9 , 27 ] on a variety of representative tasks in both regression and classification . We experiment with several real-world datasets from diverse domains , such as computer vision , tabular data , and chemical property data . Experiments demonstrate that CPT outperforms state-of-the-art methods with higher accuracy and smaller size . Our main contributions include : 1 ) We propose an interpretable generalization to the family of oblique decision trees models ; 2 ) We regularize the expressive power of CPT , using a nonparametric Bayesian shrinkage prior for each node split function ; 3 ) We provide two scalable and differentiable ways of learning CPT models , one for classification and the other for regression , which efficiently search for the optimal tree ; 4 ) We experimentally evaluate CPT on several different types of predictive tasks , illustrating that this new approach outperforms the prior work in having higher accuracy achieved with a smaller size . 2 Related work . Most decision tree literature focuses on training a single or an ensemble of trees [ 18 ] instead of making decision boundaries more flexible . One reason for this lack of research is the increase in the problem ’ s computational complexity , even for simple generalizations of the decision boundary . For example , with N as the number of data points and d as the dimension of the input space , generalizing the coordinate-wise to an oblique hyperplane cut , increases the number of possible splits of data points from Nd to Pd i=0 N i just for a single node [ 44 ] . To have more flexible decision boundaries , some methods perform hyperplane cuts in an extended feature space , created by concatenating the original and newly generated features [ 1 ] . These new features are engineered or kernel-based and not designed for interpretability . Therefore unlike CPT , they are not an interpretable extension of the decision tree . We follow this section with a literature review of the recent training algorithms for axis-aligned and oblique trees . Conventional methods for decision tree induction are greedy , where they grow the tree nodes one at a time . The greedy construction of oblique trees can be done by using coordinate descent to learn the parameters of each split [ 34 ] , or by a projection of the feature space to a lower dimension then using coordinate-cut [ 33 , 46 ] . However , the greedy procedure often leads to sub-optimal trees . There have been several attempts to non-greedy optimization , which rely on either fuzzy or probabilistic split functions [ 22 , 26 , 40 , 41 ] . The probabilistic trees are sometimes referred to as soft decision trees [ 17 , 20 , 26 ] . In these methods , the assignment of a single sample to the leaf is fuzzy or probabilistic , and gradient descent is used to optimize the tree . Most of these algorithms remain probabilistic at test time , leading to uninterpretable models as the prediction for each sample will be based on multiple leaves of the tree instead of just one . Other advances towards the training of an oblique tree are based on constructing neural networks that reproduce decision trees [ 28 , 48 ] . Yang et al . [ 48 ] use a neural network with argmax activations for the representation of classic decision tree with coordinate cuts , but they are not scalable to highdimensional data . Lee and Jaakkola [ 28 ] use the gradient of a ReLU network with a single hidden unit at each layer and skip-connections to construct an oblique decision tree . They achieve state-of-the-art results on some datasets but have to make complete trees with a high depth , consequently having a large number of leaves . In contrast to our method , there are other training algorithms that require the tree ’ s structure at the beginning . Some of the works in this direction like Bennett [ 4 ] and Bertsimas and Dunn [ 5 ] use linear programming , or mixed-integer linear programming , to find an optimum global tree . Therefore , these methods are computationally expensive and not scalable . Norouzi et al . [ 35 ] derive a convex-concave upper bound on the tree ’ s empirical loss and optimize that loss using SGD . A recent work [ 9 ] proposes tree alternating optimization , where one directly optimizes the misclassification error over separable subsets of nodes , achieving the state-of-the-art empirical performance on several datasets in a comprehensive comparison [ 49 ] . We conclude this section by connecting our splitting rule at each internal node to relevant classification algorithms [ 2 , 23 , 30 , 31 , 45 , 51 ] . The two most related works are convex polytope machine ( CPM ) [ 23 ] and infinite support hyperplane machine ( iSHM ) [ 51 ] , which both exploit the idea of learning a convex polytope associated decision boundary . iSHM is like a single hidden layer NN and is restricted to the binary classification task . 3 Convex Polytope Tree and Its Inference Algorithms . Suppose we are given the training data ( X , Y ) = { ( xn , yn ) } Nn=1 , where pairs of ( xn , yn ) are drawn independently from an identical and unknown distribution D. Each xn 2 Rd is a d-dimensional data with a corresponding label yn 2 Y . In the classification setting , Y = { 1 , · · · , K } and in regression scenario Y = R. The aim is to learn a function F : Rd ! Y that will perform well in predicting the label on samples from D. Decision tree methods construct the function F by recursively partitioning the feature space to yield a number of hierarchical , disjoint regions and assign a single label ( value ) to each region . In what follows , we will explain how we move beyond the oblique decision trees . 3.1 Convex Polytope Constrained Decision Boundary . By extending the idea of disjunctive interaction ( noisy-OR , probabilistic-OR ) [ 39 , 21 , 50 , 51 ] from probabilistic reasoning to the decision tree problem , we make the decision boundaries more flexible while preserving interpretability . To that end , we transform the problem of splitting at each node ( right or left ) to a committee of experts that make individual binary decisions ( “ Yes ” or “ No ” ) . The committee votes “ Yes ” if and only if at least one expert votes “ Yes , ” otherwise it votes “ No. ” Thus , the final vote at each node is vote = _K i=1 votei , where W denotes the logical OR operator . We model each expert as a linear classifier who votes “ Yes ” with probability P ( votei = “ Yes ” | { ri , i } , x ) = 1 ( 1 + e 0 ix ) ri , ( 1 ) where ri 0 and i 2 Rd are parameters of expert i . Now assuming that each expert votes independently , we can express the probability of the committee voting “ Yes ” as P ( vote = “ Yes ” | { ri , i } i , x ) = 1 QK i=1 ( 1 pi ) = 1 e PK i=1 ri ln ( 1+e 0ix ) , ( 2 ) where pi is the probability of expert i voting “ Yes ” and K is the total number of experts . We can now define the split function at each node by thresholding the committee voting probability : Aleft : = { x |x 2 Rd , P ( vote = “ Yes ” | { ri , i } i , x ) qthr } ( 3 ) where Aleft and Aright : = Rd \Aleft are the related splits of the space . To elaborate on the geometric shape and interpretability of the decision boundaries , consider K = 1 ( i.e. , a single expert ) . In this scenario , the decision boundary becomes a hyperplane , which is perpendicular to . In fact , the probability function for each expert is based on the signed distance of x to the hyperplane perpendicular to . Moreover , the parameter r controls how smoothly the probability transitions from 0 to 1 , where a larger r leads to sharper changes . This class of models with K = 1 and r = 1 are identical to oblique trees , which are interpretable . Furthermore , when K 1 , the interpretability is provided by the fact that linear classifiers and probabilistic-OR operation are both interpretable [ 37 ] . To geometrically analyze the implied decision regions , we provide the following theorem . Theorem 1 . For any { ri , i } Ki=1 , such that ri 2 R+ and i 2 Rd , let : Aleft = { x |x 2 Rd , f ( x ) qthr } , where : f ( x ) = 1 e PK i=1 ri ln ( 1+e 0ix ) ( 4 ) then Aleft is a convex set , confined by a convex polytope . The proof is provided in the Appendix . The above theorem shows that for K 1 , the decision region ( Aleft ) is a convex set confined by a convex K-sided polytope . More precisely , each facet of the convex polytope is a hyperplane corresponding to an expert perpendicular to its . Also worth noting , an expert with a larger r has more effect on the decision boundary , making sharper changes to the probability function . This can also be perceived as the value of their decision in the committee . Therefore , our method not only has a strong relationship with probabilistic-OR type models that provide interpretability for the model parameters [ 3 ] but also has decision boundaries with interpretable geometric characteristics . We propose a class of models , Convex Polytope Trees ( CPT ) , where each tree node follows the above splitting function .

Summary: Authors develop a generalization of oblique trees, convex polytope trees. They are able to implement them in a differentiable, end-to-end, scalable way. In their formulation. Authors compare against competitive baselines and obtain state of the art results