MANIFOLD FORESTS: CLOSING THE GAP ON NEURAL NETWORKS

1 INTRODUCTION . Decision forests , including random forests and gradient boosting trees , have solidified themselves in the past couple decades as a powerful ensemble learning method in supervised settings ( Fernández-Delgado et al. , 2014 ; Caruana & Niculescu-Mizil , 2006 ) , including both classification and regression ( Hastie et al. , 2001 ) . In classification , each forest is a collection of decision trees whose individual classifications of a data point are aggregated together using majority vote . One of the strengths of this approach is that each decision tree need only perform better than chance for the forest to be a strong learner , given a few assumptions ( Schapire , 1990 ; Biau et al. , 2008 ) . Additionally , decision trees are relatively interpretable because they can provide an understanding of which features are most important for correct classification ( Breiman , 2001 ) . Breiman originally proposed decision trees that partition the data set using hyperplanes aligned to feature axes ( Breiman , 2001 ) . Yet , this limits the flexibility of the forest and requires deep trees to classify some data sets , leading to overfitting . He also suggested that algorithms which partition based on sparse linear combinations of the coordinate axes can improve performance ( Breiman , 2001 ) . More recently , Sparse Projection Oblique Randomer Forest ( SPORF ) , partitions a random projection of the data and has shown impressive improvement over other methods ( Tomita et al. , 2015 ) . Yet random forests and other machine learning algorithms frequently operate in a tabular setting , viewing an observation ~x = ( x1 , . . . , xp ) T ∈ Rp as an unstructured feature vector . In doing so , they neglect the indices in settings where the indices encode additional information . For structured data , e.g . images or time series , traditional decision forests are not able to incorporate known continuity between features to learn new features . For decision forests to utilize known local structure in data , new features encoding this information must be manually constructed . Prior research has extended random forests to a variety of computer vision tasks ( Lepetit et al. , 2005 ; Gall et al. , 2011 ; Bosch et al. , 2007 ; Shotton et al. , 2011 ) and augmented random forests with structured pixel label information ( Kontschieder et al. , 2011 ) . Yet these methods either generate features a priori from individual pixels , and thus do not take advantage of the local topology , or lack the flexibility to learn relevant patches . Decision forests have been used to learn distance metrics on unknown manifolds ( Criminisi et al. , 2012 ) , but such manifold forest algorithms are unsupervised and aim to learn a low dimensional representation of the data . Inspired by SPORF , we propose a projection distribution that takes into account continuity between neighboring features while incorporating enough randomness to learn relevant projections . At each node in the decision tree , sets of random spatially contiguous features are randomly selected using knowledge of the underlying manifold . Summing the intensities of the sampled features yields a set of projections which can then be evaluated to partition the observations . We describe this proposed classification algorithm , Manifold Forests ( MORF ) in detail and show its effectiveness in three simulation settings as compared to common classification algorithms . Furthermore , the optimized and parallelizable open source implementation of MORF in R and Python is available . This addition makes for an effective and flexible learner across a wide range of manifold structures . 2 BACKGROUND AND RELATED WORK . 2.1 CLASSIFICATION . In the two-class classification setting , there is a data set D = { ( xi , yi ) } ni=1 of n pairs ( xi , yi ) drawn from an unknown distribution FXY where xi ∈ X ⊂ Rp and yi ∈ Y = { 0 , 1 } . Our goal is to train a classifier h ( x ; Dn ) : X× ( X×Y ) n → Y based on our observations that generalizes to correctly predict the class of an observed x . The performance of this classifier is evaluated via the 0-1 Loss function L ( h ( x ) , y ) = I [ h ( x ) 6= y ] to find the optimal classifier h∗ = argminh E [ L ( h ( x ) , y ) ] , which minimizes the probability of an incorrect classification . 2.2 RANDOM FORESTS . Originally popularized by Breiman , the random forest ( RF ) classifier is empirically very effective ( Fernández-Delgado et al. , 2014 ) while maintaining strong theoretical guarantees ( Breiman , 2001 ) . A random forest is an ensemble of decision trees whose individual classifications of a data point are aggregated together using majority vote . Each decision tree consists of split nodes and leaf nodes . A split node is associated with a subset of the data S = { ( xi , yi ) } ⊆ D and splits into two child nodes , each associated with a binary partition of S. Let ej ∈ Rp denote a unit vector in the standard basis ( that is , a vector with a single one and the rest of the entries are zero ) and τ a threshold value . Then S is partitioned into two subsets given the pair θj = { ej , τ } . SLθ = { ( xi , yi ) | eTj xi < τ } SRθ = { ( xi , yi ) | eTj xi ≥ τ } To choose the partition , the optimal θ∗ = ( e∗j , τ ∗ ) pair is selected via a greedy search from among a set of d randomly selected standard basis vectors ej . The selected partition is that which maximizes some measure of information gain . A typical measure is a decrease in impurity , calculated by the Gini impurity score I ( S ) , of the resulting partitions ( Hastie et al. , 2001 ) . Let p̂k = 1|S| ∑ yi∈S I [ yi = k ] be the fraction of elements of class k in partition S , then the optimal split is found as I ( S ) = K∑ k=1 p̂k ( 1− p̂k ) θ∗ = argmax θ |S|I ( S ) − |SLθ |I ( SLθ ) − |SRθ |I ( SRθ ) . A leaf node is created once the partition reaches a stopping criterion , typically either falling below an impurity score threshold or a minimum number of observations ( Hastie et al. , 2001 ) . The leaf nodes of the tree form a disjoint partition of the feature space in which each partition of observations Sk is assigned a class label c∗k corresponding to the class majority . c∗k = argmax ck ∑ yi∈S I ( yi = ck ) . A decision tree classifies a new observation by assigning it the class of the partition into which the observation falls . The forest averages the classifications over all decision trees to make the final classification ( Hastie et al. , 2001 ) . For good performance of the ensemble and strong theoretical guarantees , the individual decision trees must be relatively uncorrelated from one another . Breiman ’ s random forest algorithm does this in two ways : 1 . At every node in the decision tree , the optimal split is determined over a random subset d of the total collection of features p. 2 . Each tree is trained on a randomly bootstrapped sample of data points D′ ⊂ D from the full training data set . Applying these techniques means that random forests do not overfit and lowers the upper bound of the generalization error ( Breiman , 2001 ) . 2.3 SPARSE PROJECTION OBLIQUE RANDOMER FORESTS . SPORF is a recent modification to random forest that has shown improvement over other versions ( Tomita et al. , 2015 ; Tomita et al. , 2017 ) . Recall that RF split nodes partition data along the coordinate axes by comparing the projection eTj xi of observation xi on standard basis ej to a threshold value τ . SPORF generalizes the set of possible projections , allowing for the data to be partitioned along axes specified by any sparse vector a. SLθ = { ( xi , yi ) | aTxi < τ } SRθ = { ( xi , yi ) | aTxi ≥ τ } Rather than partitioning the data solely along the coordinate axes ( i.e . the standard basis ) , SPORF creates partitions along axes specified by sparse vectors . In other words , let the dictionary A be the set of atoms { a } , each atom a p-dimensional vector defining a possible projection aTxi . In axis-aligned forests , A is the set of standard basis vectors { ej } . In SPORF , the dictionary D can be much larger , because it includes , for example , all 2-sparse vectors . At each split node , SPORF samples d atoms from D according to a specified distribution . By default , each of the d atoms are randomly generated with a sparsity level drawn from a Poisson distribution with a specified rate λ . Then , each of the non-zero elements are uniformly randomly assigned either +1 or −1 . Note that the size of the dictionary for SPORF is 3p ( because each of the p elements could be −1 , 0 , or +1 ) , although the atoms are sampled from a distribution heavily skewed towards sparsity . 3 METHODS . 3.1 RANDOM PROJECTION FORESTS ON MANIFOLDS . In the structured setting , the dictionary of projection vectors A = { a } is modified to take advantage of the underlying manifold on which the data lies . We term this method the Manifold Forest ( MORF ) . Each atom a projects an observation to a real number and is designed with respect to prior knowledge of the data manifold . Nonzero elements of a effectively select and weight features . Since the feature space is structured , each element of a maps to a location on the underlying manifold . Thus , patterns of contiguous points on the manifold define the atoms of A ; the distribution of those patterns yields a distribution over the atoms . At each node in the decision tree , MORF samples d atoms , yielding d new features per observation . MORF proceeds just like SPORF by optimizing the best split according to the Gini index . Algorithm pseudocode , essentially equivalent to that of SPORF , can be found in the Appendix . In the case of two-dimensional arrays , such as images , an observation xi ∈ Rp is a vectorized representation of a data-matrix Xi ∈ RW×H . To capture the relevance of neighboring pixels , MORF creates projections by summing the intensities of pixels in rectangular patches . Thus the atoms of A are the vectorized representations of these rectangular patches . A rectangular patch is fully parameterized by the location of its upper-left corner ( u , v ) , its height h , and width w. To generate a patch , first the index of the upper left corner is uniformly sampled . Then its height and width are independently sampled from separate uniform distributions . MORF hyperparameters determine the minimum and maximum heights heights { hmin , hmax } and widths { wmin , wmax } , respectively , to sample from . Let unif { α , β } denote the discrete uniform distribution . An atom a is sampled as follows . Note that the patch can not exceed the data-matrix boundaries . u ∼ unif { 0 , W − wmin } v ∼ unif { 0 , H − hmin } w ∼ unif { wmin , min ( wmax , W − u ) } h ∼ unif { hmin , min ( hmax , H − v ) } The vectorized atom a yields a projection of the data aTxi , effectively selecting and summing pixel intensities in the sampled rectangular patch . aT = ( 0 , . . . , 0︸ ︷︷ ︸ W×v , 0 , . . . , 0︸ ︷︷ ︸ u , ( h−1 ) ×W︷ ︸︸ ︷ 1 , . . . , 1︸ ︷︷ ︸ w , 0 , . . . , 0︸ ︷︷ ︸ W−w , . . . , 1 , . . . , 1︸ ︷︷ ︸ w , 0 , . . . , 0︸ ︷︷ ︸ W−w , 1 , . . . , 1︸ ︷︷ ︸ w , 0 , . . . , 0︸ ︷︷ ︸ W−w−u , 0 , . . . , 0︸ ︷︷ ︸ W× ( H−h−v ) ) By constructing features in this way , MORF learns low-level features in the structured data , such as edges or corners in images . The forest can therefore learn the features that best distinguish a class . The structure of these atoms is flexible and task dependent . In the case of data lying on a cyclic manifold , the atoms can wrap-around borders to capture the added continuity . Atoms can also be used in one-dimensional arrays , such as univariate time-series data , in which case hmin = hmax = 1

Summary: This paper proposed a new method called manifold forest to improve decision forest (DF) classification results. It is motivated by that, natural data is often in some manifold but not randomly distributed. It showed how to use the 2D spatial structures of natural images by constructing structured atoms. Results on 3 toy examples and MNIST showed the better performance than standard RF and SPORF.