Stochastic Induction of Decision Trees with Application to Learning Haar Tree

Decision trees are a convenient and established approach for any supervised learning task . Decision trees are used in a broad range of applications from medical imaging to computer vision . Decision trees are trained by greedily splitting the leaf nodes into a split and two leaf nodes until a certain stopping criterion is reached . The procedure of splitting a node consists of finding the best feature and threshold that minimizes a criterion . The criterion minimization problem is solved through an exhaustive search algorithm . However , this exhaustive search algorithm is very expensive , especially , if the number of samples and features are high . In this paper , we propose a novel stochastic approach for the criterion minimization . Asymptotically , the proposed algorithm is faster than conventional exhaustive search by several orders of magnitude . It is further shown that the proposed approach minimizes an upper bound for the criterion . Experimentally , the algorithm is compared with several other related state-of-the-art decision tree learning methods , including the baseline non-stochastic approach . The proposed algorithm outperforms every other decision tree learning ( including online and fast ) approaches and performs as well as the baseline algorithm in terms of accuracy and computational cost , despite being non-deterministic . For empirical evaluation , we apply the proposed algorithm to learn a Haar tree over MNIST dataset that consists of over 200 , 000 features and 60 , 000 samples . This tree achieved a test accuracy of 94 % over MNIST which is 4 % higher than any other known axis-aligned tree . This result is comparable to the performance of oblique trees , while providing a significant speed-up at both inference and training times . 1 INTRODUCTION . Decision trees are among the most established machine learning models in the scientific community and have been widely used in various applications , such as computer vision and medical imaging Criminisi & Shotton ( 2013 ) ; Freund et al . ( 1999 ) ; Breiman ( 2001 ) ; Tavallali et al . ( 2019 ) . Decision trees are extensively used as a sub-procedure in ensemble models , such as boosting and forest Criminisi & Shotton ( 2013 ) . Decision tree training consists of recursively splitting a leaf node into one split and two leaf nodes Breiman et al . ( 1984 ) ; Gehrke et al . ( 1999 ) . The split node consists of a decision stump that routs the samples to either of its left or right children . The splitting of a leaf node is cast as selecting the best feature and threshold that minimizes a desired criterion . The criterion is an impurity function , such as Gini index Breiman et al . ( 1984 ) , cross-entropy Quinlan ( 1986 ) , uniform piece-wise constant densities Ram & Gray ( 2011 ) , Gaussian differential ( continuous ) entropy Criminisi & Shotton ( 2013 ) and others Yang & Wong ( 2014 ) ; Liu & Wong ( 2014 ) ; Sheikhi & Babamir ( 2018 ; 2016 ) . The best feature and threshold are selected through an incremental algorithm that evaluates combination of every possible feature and threshold . However , this algorithm becomes computationally very expensive if the dataset is large or high dimensional . The computational complexity of solving criterion minimization problem isO ( DNlogN ) , whereN and D are number of samples and dimensionality , respectively . The computational complexity becomes very high especially when N and D are high since N and D are multiplied in the computational complexity term ( O ( DNlogN ) ) . In this paper , we tackle the abovementioned issue by applying the splitting algorithm in a novel iterative and stochastic fashion , where less important features are discarded at each iteration . The literature includes many proposals for speeding up the training procedures of various learning algorithms . One common approach is preprocessing the dataset by reducing the sample size or applying some feature selection technique Dash & Liu ( 1997 ) ; Jović et al . ( 2015 ) ; John et al . ( 1994 ) . Pre-training feature selection will potentially lead to a less computationally expensive process for learning Jović et al . ( 2015 ) ; Kohavi & Sommerfield ( 1995 ) ; Koller & Sahami ( 1996 ) ; Korn et al . ( 2001 ) . Feature selection algorithms are generally categorized in two groups of filter and wrapper approaches . The filter approaches consist of selecting features with regard to a desired criterion Jović et al . ( 2015 ) . Generally , filter approaches use statistical measures , such as minimum Redundancy maximum Relevance- ( mRmR ) Tang et al . ( 2014 ) , correlation Yu & Liu ( 2003 ) , chi-square Moh ’ d A Mesleh ( 2007 ) , information gain Bhattacharyya & Kalita ( 2013 ) , gain ratio Witten & Frank ( 2002 ) , etc . The wrapper approaches consist of selecting features and minimizing the loss function jointly Bhattacharyya & Kalita ( 2013 ) ; Bradley & Mangasarian ( 1998 ) ; Maldonado et al . ( 2014 ) ; Hastie et al . ( 2009 ) . However , feature selection as a preprocessing method might not lead to an accurate final model since features are not guaranteed to contribute to lower loss Jović et al . ( 2015 ) . Nevertheless , wrappers are practical only for fast modeling algorithms or greedy search strategies , such as linear SVM Liu et al . ( 2014 ) , Nave Bayes Cortizo & Giraldez ( 2006 ) , and Extreme Learning Machines Benoı̂t et al . ( 2013 ) . In Embedded approaches , first a filter approach is applied to select a subset of features and then followed by a wrapper approach to select the best candidate feature Guyon & Elisseeff ( 2003 ) ; Das ( 2001 ) . Different decision tree algorithms , such as CART Breiman et al . ( 1984 ) , C4.5 Quinlan ( 2014 ) , and random forest are commonly used as embedded methods Sandri & Zuccolotto ( 2006 ) . The feature selection as a preprocessing approach decreases the computational complexity of training algorithm . However , it is suboptimal w.r.t . the objective function of training , and is inefficient for large datasets since it does not decrease computation over number of samples Rodriguez-Lujan et al . ( 2010 ) . A well-known approach that tackles the issue of large datasets is online learning Utgoff ( 1989 ) . Various studies consider applying online learning to decision trees Utgoff ( 1989 ) ; Schlimmer & Fisher ( 1986 ) ; Wang et al . ( 2003 ) . Such algorithms are fast , simple , and often make few statistical assumptions . The objective of online learning is to update and train the model from a stream of data Shalev-Shwartz et al . ( 2011 ) . At each time step , the model gets updated optimally with respect to the recently acquired samples . Online learning is more computationally efficient compared to offline learning where the entire trainset is used for training Shalev-Shwartz et al . ( 2011 ) . However , these algorithms increase the size of the tree when visiting a new batch of the data . Despite the extensive usages and applications of decision trees Criminisi & Shotton ( 2013 ) , few algorithms have focused on decreasing the high computational cost of inducing a tree ? ? . However , the literature leaves a gap in treating the high computational complexity of inducing the decision tree and exploring potential improvements . None of the previous research propose the computational complexity of their algorithm or provide any analysis of the goodness of features selected at the split node . To address this gap , we investigated both computational complexity of SDT and its efficiency in finding near optimum feature at the split node . In ? , authors propose to randomly select a subset of features at the split node and then find the best feature among the set using all samples present at the node . However , due to randomness , the optimality of this process is not guaranteed . The high computational cost is mainly rooted in the exhaustive search algorithm applied for solving the split criterion . This algorithm specially becomes computationally expensive if the dataset is large and high dimensional . It is because the computational complexity of solving the splitting criterion is O ( DNlogN ) Hoare ( 1961 ) ; Breiman et al . ( 1984 ) . To counter the aforementioned issues , we propose an accurate and fast Stochastic Decision Tree ( SDT ) induction algorithm that minimizes the splitting criterion efficiently . At a node , the proposed algorithm starts with an empty set St and all D features . At node j , the algorithm iteratively selects a tiny random set of samples from Sj ( in order of 2−C × |Sj | ) and discards half of the less important features w.r.t St. Sj and |Sj | are the set and number of samples present at the jth node , respectively . For the best feature and threshold found at each iteration , SDT minimizes an upper bound of the splitting criterion , monotonically . Overall , the algorithm steeply descends towards the minimum of the criterion function . It is also demonstrated by the results at the experiments section . Mathematically speaking , we show that the algorithm provides higher priority to more distinctive features . Essentially , the more distinctive a feature , the greater the chance of it being selected at the final iteration . The computational complexity of the algorithm for constructing a tree of depth ∆ is O ( ∆DNlog N 2C 2C ) . C is a hyperparameter and in our experiments we set it to 10 . The computational complexity achieved by SDT is smaller than that of the original approach ( withO ( ∆DNlogN ) ) and its optimal implementation ( O ( ∆DN +DNlogN ) ) by several orders of magnitude . Our approach is faster because it focuses on breaking down the computational complexity created due to the large number of samples and features . Numerically speaking , SDT shows its advantage over other implementations when the sizes of samples and features are high ( ND is very high ) . Experimentally , the proposed algorithm could achieve same or improved accuracy as the original greedy algorithm Breiman et al . ( 1984 ) while surpassing every other related state-of-the-art algorithms both in terms of accuracy and complexity by several orders of magnitude . For empirical evaluation , we applied our algorithm to learn a Haar tree over MNIST where Haar Viola & Jones ( 2001 ) filters were used to extract 200 , 000 features from the dataset . The Haar tree could achieve competitive accuracy to more complicated trees , such as oblique trees and optimal oblique trees while its inference time was far smaller . Theoretically , an oblique tree needs O ( D∆ ) operations to produce prediction , while Haar tree only needs O ( D + ∆ ) . ∆ is the maximum depth of the tree . In terms of size , Haar tree takes O ( 2∆ ) space while oblique tree takes O ( D2∆ ) space . 2 PRELIMINARIES . 2.1 INDUCTION OF A DECISION TREE . Every decision tree construction algorithm tends to minimize the following objective function Lα ( T ) = L ( T ) + α× leaves ( T ) , α > 0 , ( 1 ) where , L ( T ) is a loss function of the tree ( T ) over a trainset , leaves ( . ) represents the number of leaves in a tree , and α > 0 is cost complexity coefficient . After constructing the tree , pruning is performed to minimize the cost function Lα ( T ) over the validationset . Finding the optimum of this problem is NP-complete ? . Therefore , state-of-the-art algorithms for constructing a tree consist of greedily splitting a leaf node into a split and two leaf nodes . Assume a dataset S consisting N pairs of observation { ( xi , yi ) } Ni=1 where xi ∈ RD and yi ∈ { 1 , 2 , 3 , ... , M } . The function fj ( x ) at the jth split node consists of a decision stump fj ( x ) = sign ( xp − th ) where superscript p and th represent pth feature and the threshold , respectively . At the growing phase of a decision tree , p and th are found optimally with respect to some criteria Breiman et al . ( 1984 ) . Usually the tree is grown until some stopping criterion or maximum depth is reached . The problem of constructing a split node consist of minimizing the desired criterion over the samples of each child of the node . min fj ∑ fj= { −1,1 } |Sfjj | |Sj | H ( S fj j ) s.t fj ( x ) = sign ( x p − th ) ( 2 ) where , Sfjj is set of samples at the left child ( fj = −1 ) or right child ( fj = 1 ) of a node j , Sj is the set of samples present at the jth node , and |.| returns the number of samples at its input set . H is an entropy function . The optimum of problem equation 2 can be found in O ( DNj logNj ) where Nj and D are number of samples at node j ( |Sj | ) and features , respectively . This is done by sorting all samples along each feature and incrementally finding the best threshold for each feature Breiman et al . ( 1984 ) .

Summary: The paper introduces an approach to decision tree induction based on sub-sampling and pruning heuristics.  Typically, inducing a tree involves O(D*N_j*log(N_j)) work at each node j, where D is the number of features and N_j is the number of observations at node j.  This is because we can find the optimal split at node j for each feature in O(N_j) time after sorting each of the N_j observations. The basic idea introduced in this paper is to reduce the computational complexity incurred at each node by starting with a subsample of N_j, pruning features with sub-par performance, increasing the subsample size, pruning sub-par performing features, .... until the best feature has been chosen on the full sample of N_j observations.  The idea is very simple and the authors attempt to bolster this simple idea with some theoretical results and a wide array of experiments. The method does appear to be effective for datasets with very large D, which for example occurs when constructing Haar features on image data.