Stochastic Induction of Decision Trees with Application to Learning Haar Tree

Decision trees are a convenient and established approach for any supervised learning task . Decision trees are used in a broad range of applications from medical imaging to computer vision . Decision trees are trained by greedily splitting the leaf nodes into a split and two leaf nodes until a certain stopping criterion is reached . The procedure of splitting a node consists of finding the best feature and threshold that minimizes a criterion . The criterion minimization problem is solved through an exhaustive search algorithm . However , this exhaustive search algorithm is very expensive , especially , if the number of samples and features are high . In this paper , we propose a novel stochastic approach for the criterion minimization . Asymptotically , the proposed algorithm is faster than conventional exhaustive search by several orders of magnitude . It is further shown that the proposed approach minimizes an upper bound for the criterion . Experimentally , the algorithm is compared with several other related state-of-the-art decision tree learning methods , including the baseline non-stochastic approach . The proposed algorithm outperforms every other decision tree learning ( including online and fast ) approaches and performs as well as the baseline algorithm in terms of accuracy and computational cost , despite being non-deterministic . For empirical evaluation , we apply the proposed algorithm to learn a Haar tree over MNIST dataset that consists of over 200 , 000 features and 60 , 000 samples . This tree achieved a test accuracy of 94 % over MNIST which is 4 % higher than any other known axis-aligned tree . This result is comparable to the performance of oblique trees , while providing a significant speed-up at both inference and training times . 1 INTRODUCTION . Decision trees are among the most established machine learning models in the scientific community and have been widely used in various applications , such as computer vision and medical imaging Criminisi & Shotton ( 2013 ) ; Freund et al . ( 1999 ) ; Breiman ( 2001 ) ; Tavallali et al . ( 2019 ) . Decision trees are extensively used as a sub-procedure in ensemble models , such as boosting and forest Criminisi & Shotton ( 2013 ) . Decision tree training consists of recursively splitting a leaf node into one split and two leaf nodes Breiman et al . ( 1984 ) ; Gehrke et al . ( 1999 ) . The split node consists of a decision stump that routs the samples to either of its left or right children . The splitting of a leaf node is cast as selecting the best feature and threshold that minimizes a desired criterion . The criterion is an impurity function , such as Gini index Breiman et al . ( 1984 ) , cross-entropy Quinlan ( 1986 ) , uniform piece-wise constant densities Ram & Gray ( 2011 ) , Gaussian differential ( continuous ) entropy Criminisi & Shotton ( 2013 ) and others Yang & Wong ( 2014 ) ; Liu & Wong ( 2014 ) ; Sheikhi & Babamir ( 2018 ; 2016 ) . The best feature and threshold are selected through an incremental algorithm that evaluates combination of every possible feature and threshold . However , this algorithm becomes computationally very expensive if the dataset is large or high dimensional . The computational complexity of solving criterion minimization problem isO ( DNlogN ) , whereN and D are number of samples and dimensionality , respectively . The computational complexity becomes very high especially when N and D are high since N and D are multiplied in the computational complexity term ( O ( DNlogN ) ) . In this paper , we tackle the abovementioned issue by applying the splitting algorithm in a novel iterative and stochastic fashion , where less important features are discarded at each iteration . The literature includes many proposals for speeding up the training procedures of various learning algorithms . One common approach is preprocessing the dataset by reducing the sample size or applying some feature selection technique Dash & Liu ( 1997 ) ; Jović et al . ( 2015 ) ; John et al . ( 1994 ) . Pre-training feature selection will potentially lead to a less computationally expensive process for learning Jović et al . ( 2015 ) ; Kohavi & Sommerfield ( 1995 ) ; Koller & Sahami ( 1996 ) ; Korn et al . ( 2001 ) . Feature selection algorithms are generally categorized in two groups of filter and wrapper approaches . The filter approaches consist of selecting features with regard to a desired criterion Jović et al . ( 2015 ) . Generally , filter approaches use statistical measures , such as minimum Redundancy maximum Relevance- ( mRmR ) Tang et al . ( 2014 ) , correlation Yu & Liu ( 2003 ) , chi-square Moh ’ d A Mesleh ( 2007 ) , information gain Bhattacharyya & Kalita ( 2013 ) , gain ratio Witten & Frank ( 2002 ) , etc . The wrapper approaches consist of selecting features and minimizing the loss function jointly Bhattacharyya & Kalita ( 2013 ) ; Bradley & Mangasarian ( 1998 ) ; Maldonado et al . ( 2014 ) ; Hastie et al . ( 2009 ) . However , feature selection as a preprocessing method might not lead to an accurate final model since features are not guaranteed to contribute to lower loss Jović et al . ( 2015 ) . Nevertheless , wrappers are practical only for fast modeling algorithms or greedy search strategies , such as linear SVM Liu et al . ( 2014 ) , Nave Bayes Cortizo & Giraldez ( 2006 ) , and Extreme Learning Machines Benoı̂t et al . ( 2013 ) . In Embedded approaches , first a filter approach is applied to select a subset of features and then followed by a wrapper approach to select the best candidate feature Guyon & Elisseeff ( 2003 ) ; Das ( 2001 ) . Different decision tree algorithms , such as CART Breiman et al . ( 1984 ) , C4.5 Quinlan ( 2014 ) , and random forest are commonly used as embedded methods Sandri & Zuccolotto ( 2006 ) . The feature selection as a preprocessing approach decreases the computational complexity of training algorithm . However , it is suboptimal w.r.t . the objective function of training , and is inefficient for large datasets since it does not decrease computation over number of samples Rodriguez-Lujan et al . ( 2010 ) . A well-known approach that tackles the issue of large datasets is online learning Utgoff ( 1989 ) . Various studies consider applying online learning to decision trees Utgoff ( 1989 ) ; Schlimmer & Fisher ( 1986 ) ; Wang et al . ( 2003 ) . Such algorithms are fast , simple , and often make few statistical assumptions . The objective of online learning is to update and train the model from a stream of data Shalev-Shwartz et al . ( 2011 ) . At each time step , the model gets updated optimally with respect to the recently acquired samples . Online learning is more computationally efficient compared to offline learning where the entire trainset is used for training Shalev-Shwartz et al . ( 2011 ) . However , these algorithms increase the size of the tree when visiting a new batch of the data . Despite the extensive usages and applications of decision trees Criminisi & Shotton ( 2013 ) , few algorithms have focused on decreasing the high computational cost of inducing a tree ? ? . However , the literature leaves a gap in treating the high computational complexity of inducing the decision tree and exploring potential improvements . None of the previous research propose the computational complexity of their algorithm or provide any analysis of the goodness of features selected at the split node . To address this gap , we investigated both computational complexity of SDT and its efficiency in finding near optimum feature at the split node . In ? , authors propose to randomly select a subset of features at the split node and then find the best feature among the set using all samples present at the node . However , due to randomness , the optimality of this process is not guaranteed . The high computational cost is mainly rooted in the exhaustive search algorithm applied for solving the split criterion . This algorithm specially becomes computationally expensive if the dataset is large and high dimensional . It is because the computational complexity of solving the splitting criterion is O ( DNlogN ) Hoare ( 1961 ) ; Breiman et al . ( 1984 ) . To counter the aforementioned issues , we propose an accurate and fast Stochastic Decision Tree ( SDT ) induction algorithm that minimizes the splitting criterion efficiently . At a node , the proposed algorithm starts with an empty set St and all D features . At node j , the algorithm iteratively selects a tiny random set of samples from Sj ( in order of 2−C × |Sj | ) and discards half of the less important features w.r.t St. Sj and |Sj | are the set and number of samples present at the jth node , respectively . For the best feature and threshold found at each iteration , SDT minimizes an upper bound of the splitting criterion , monotonically . Overall , the algorithm steeply descends towards the minimum of the criterion function . It is also demonstrated by the results at the experiments section . Mathematically speaking , we show that the algorithm provides higher priority to more distinctive features . Essentially , the more distinctive a feature , the greater the chance of it being selected at the final iteration . The computational complexity of the algorithm for constructing a tree of depth ∆ is O ( ∆DNlog N 2C 2C ) . C is a hyperparameter and in our experiments we set it to 10 . The computational complexity achieved by SDT is smaller than that of the original approach ( withO ( ∆DNlogN ) ) and its optimal implementation ( O ( ∆DN +DNlogN ) ) by several orders of magnitude . Our approach is faster because it focuses on breaking down the computational complexity created due to the large number of samples and features . Numerically speaking , SDT shows its advantage over other implementations when the sizes of samples and features are high ( ND is very high ) . Experimentally , the proposed algorithm could achieve same or improved accuracy as the original greedy algorithm Breiman et al . ( 1984 ) while surpassing every other related state-of-the-art algorithms both in terms of accuracy and complexity by several orders of magnitude . For empirical evaluation , we applied our algorithm to learn a Haar tree over MNIST where Haar Viola & Jones ( 2001 ) filters were used to extract 200 , 000 features from the dataset . The Haar tree could achieve competitive accuracy to more complicated trees , such as oblique trees and optimal oblique trees while its inference time was far smaller . Theoretically , an oblique tree needs O ( D∆ ) operations to produce prediction , while Haar tree only needs O ( D + ∆ ) . ∆ is the maximum depth of the tree . In terms of size , Haar tree takes O ( 2∆ ) space while oblique tree takes O ( D2∆ ) space . 2 PRELIMINARIES . 2.1 INDUCTION OF A DECISION TREE . Every decision tree construction algorithm tends to minimize the following objective function Lα ( T ) = L ( T ) + α× leaves ( T ) , α > 0 , ( 1 ) where , L ( T ) is a loss function of the tree ( T ) over a trainset , leaves ( . ) represents the number of leaves in a tree , and α > 0 is cost complexity coefficient . After constructing the tree , pruning is performed to minimize the cost function Lα ( T ) over the validationset . Finding the optimum of this problem is NP-complete ? . Therefore , state-of-the-art algorithms for constructing a tree consist of greedily splitting a leaf node into a split and two leaf nodes . Assume a dataset S consisting N pairs of observation { ( xi , yi ) } Ni=1 where xi ∈ RD and yi ∈ { 1 , 2 , 3 , ... , M } . The function fj ( x ) at the jth split node consists of a decision stump fj ( x ) = sign ( xp − th ) where superscript p and th represent pth feature and the threshold , respectively . At the growing phase of a decision tree , p and th are found optimally with respect to some criteria Breiman et al . ( 1984 ) . Usually the tree is grown until some stopping criterion or maximum depth is reached . The problem of constructing a split node consist of minimizing the desired criterion over the samples of each child of the node . min fj ∑ fj= { −1,1 } |Sfjj | |Sj | H ( S fj j ) s.t fj ( x ) = sign ( x p − th ) ( 2 ) where , Sfjj is set of samples at the left child ( fj = −1 ) or right child ( fj = 1 ) of a node j , Sj is the set of samples present at the jth node , and |.| returns the number of samples at its input set . H is an entropy function . The optimum of problem equation 2 can be found in O ( DNj logNj ) where Nj and D are number of samples at node j ( |Sj | ) and features , respectively . This is done by sorting all samples along each feature and incrementally finding the best threshold for each feature Breiman et al . ( 1984 ) .

Summary: The paper proposes a stochastic (i.e. approximate) algorithm to fasten node splitting during tree induction. The idea of the method is to filter out the most irrelevant features using a (growing) subsample of training examples and then to optimize the split using all samples only for the remaining features. The approach is shown to improve computing times for tree construction by several orders of magnitude without impacting accuracy.