Adaptive Neural Trees

1 INTRODUCTION . Neural networks ( NNs ) and decision trees ( DTs ) are both powerful classes of machine learning models with proven successes in academic and commercial applications . The two approaches , however , typically come with mutually exclusive benefits and limitations . NNs are characterised by learning hierarchical representations of data through the composition of nonlinear transformations ( Zeiler & Fergus , 2014 ; Bengio , 2013 ) , which has alleviated the need for feature engineering , in contrast with many other machine learning models . In addition , NNs are trained with stochastic optimisers , such as stochastic gradient descent ( SGD ) , allowing training to scale to large datasets . Consequently , with modern hardware , we can train NNs of many layers on large datasets , solving numerous problems ranging from object detection to speech recognition with unprecedented accuracy ( LeCun et al. , 2015 ) . However , their architectures typically need to be designed by hand and fixed per task or dataset , requiring domain expertise ( Zoph & Le , 2017 ) . Inference can also be heavy-weight for large models , as each sample engages every part of the network , i.e. , increasing capacity causes a proportional increase in computation ( Bengio et al. , 2013 ) . Alternatively , DTs are characterised by learning hierarchical clusters of data ( Criminisi & Shotton , 2013 ) . A DT learns how to split the input space , so that in each subset , linear models suffice to explain the data . In contrast to standard NNs , the architectures of DTs are optimised based on training data , and are particularly advantageous in data-scarce scenarios . DTs also enjoy lightweight inference as only a single root-to-leaf path on the tree is used for each input sample . However , successful applications of DTs often require hand-engineered features of data . We can ascribe the limited expressivity of single DTs to the common use of simplistic routing functions , such as splitting on axis-aligned features . The loss function for optimising hard partitioning is non-differentiable , which hinders the use of gradient descent-based optimization and thus complex splitting functions . Current techniques for increasing capacity include ensemble methods such as random forests ( RFs ) ( Breiman , 2001 ) and gradient-boosted trees ( GBTs ) ( Friedman , 2001 ) , which are known to achieve state-of-the-art performance in various tasks , including medical imaging and financial forecasting ( Sandulescu & Chiru , 2016 ; Kaggle.com , 2017 ; Le Folgoc et al. , 2016 ; Volkovs et al. , 2017 ) . The goal of this work is to combine NNs and DTs to gain the complementary benefits of both approaches . To this end , we propose adaptive neural trees ( ANTs ) , which generalise previous work that attempted the same unification ( Suárez & Lutsko , 1999 ; İrsoy et al. , 2012 ; Laptev & Buhmann , 2014 ; Rota Bulo & Kontschieder , 2014 ; Kontschieder et al. , 2015 ; Frosst & Hinton , 2017 ; Xiao , 2017 ) and address their limitations ( see Tab . 1 ) . ANTs represent routing decisions and root-to-leaf computational paths within the tree structures as NNs , which lets them benefit from hierarchical representation learning , rather than being restricted to partitioning the raw data space . In addition , we propose a backpropagation-based training algorithm to grow ANTs based on a series of decisions between making the ANT deeper—the central NN paradigm—or partitioning the data—the central DT paradigm ( see Fig . 1 ( Right ) ) . This allows the architectures of ANTs to adapt to the data available . By our design , ANTs inherit the following desirable properties from both DTs and NNs : • Representation learning : as each root-to-leaf path in an ANT is a NN , features can be learnt end-to-end with gradient-based optimisation . This , in turn , allows for learning complex data partitioning . The training algorithm is also amenable to SGD . • Architecture learning : by progressively growing ANTs , the architecture adapts to the availability and complexity of data , embodying Occams razor . The growth procedure can be viewed as architecture search with a hard constraint over the model class . • Lightweight inference : at inference time , ANTs perform conditional computation , selecting a single root-to-leaf path on the tree on a per-sample basis , activating only a subset of the parameters of the model . We empirically validate these benefits for classification and regression through experiments on the MNIST ( LeCun et al. , 1998 ) , CIFAR-10 ( Krizhevsky & Hinton , 2009 ) and SARCOS ( Vijayakumar & Schaal , 2000 ) datasets . Along with other forms of neural networks , ANTs far outperform state-of-the-art random forest ( RF ) ( Zhou & Feng , 2017 ) and gradient boosted tree ( GBT ) ( Ponomareva et al. , 2017 ) methods on the image-based classification datasets , with architectures achieving over 99 % accuracy on MNIST and over 90 % accuracy on CIFAR-10 . On the other hand , the best performing methods on the SARCOS multivariate regression dataset are all tree-based , with soft decision trees ( SDTs ) ( Suárez & Lutsko , 1999 ; Jordan & Jacobs , 1994 ) , GBTs ( Friedman , 2001 ) and ANTs achieving the lowest mean squared error . At the same time , ANTs can learn meaningful hierarchical partitionings of data , e.g. , grouping man-made and natural objects ( see Fig . 2 ) . ANTs also have reduced time and memory requirements during inference , conferred by conditional computation . In one case , we discover an architecture that achieves over 98 % accuracy on MNIST using approximately the same number of parameters as a linear classifier on raw image pixels , showing the benefits of modelling a hierarchical structure that reflects the underlying data structure in enhancing both computational and predictive performance . Finally , we demonstrate the benefits of architecture learning by training ANTs on subsets of CIFAR-10 of varying sizes . The method can construct architectures of adequate size , leading to better generalisation , particularly on small datasets . 2 RELATED WORK . Our work is primarily related to research into combining DTs and NNs to benefit from the power of representation learning . Here we explain how ANTs subsumes a large body of such prior work as specific cases and address their limitations . We include additional reviews of work in conditional computation and neural architecture search in Sec . B in the supplementary material . The very first SDT introduced in ( Suárez & Lutsko , 1999 ) is a specific case where in our terminology the routers are axis-aligned features , the transformers are identity functions , and the routers are static distributions over classes or linear functions . The hierarchical mixture of experts ( HMEs ) proposed by ( Jordan & Jacobs , 1994 ) is a variant of SDTs whose routers are linear classifiers and the tree structure is fixed . More modern SDTs in ( Rota Bulo & Kontschieder , 2014 ; Laptev & Buhmann , 2014 ; Frosst & Hinton , 2017 ) used multilayer perceptrons ( MLPs ) or convolutional layers in the routers to learn more complex partitionings of the input space . However , the simplicity of identity transformers used in these methods means that input data is never transformed and thus each path on the tree does not perform representation learning , limiting their performance . More recent work suggested that integrating non-linear transformations of data into DTs would enhance model performance . The neural decision forest ( NDF ) ( Kontschieder et al. , 2015 ) , which held cutting-edge performance on ImageNet ( Deng et al. , 2009 ) in 2015 , is an ensemble of DTs , each of which is also an instance of ANTs where the whole GoogLeNet architecture ( Szegedy et al. , 2015 ) ( except for the last linear layer ) is used as the root transformer , prior to learning tree-structured classifiers with linear routers . Xiao ( 2017 ) employed a similar approach with a MLP at the root transformer , and is optimised to minimise a differentiable information gain loss . The conditional network proposed in ( Ioannou et al. , 2016 ) sparsified CNN architectures by distributing computations on hierarchical structures based on directed acyclic graphs with MLP-based routers , and designed models with the same accuracy with reduced compute cost and number of parameters . However , in all cases , the model architectures are pre-specified and fixed . In contrast , ANTs satisfy all criteria in Tab . 1 ; they provide a general framework for learning treestructured models with the capacity of representation learning along each path and within routing functions , and a mechanism for learning its architecture . Architecture growth is a key facet of DTs ( Criminisi & Shotton , 2013 ) , and typically performed in a greedy fashion with a termination criteria based on validation set error ( Suárez & Lutsko , 1999 ; İrsoy et al. , 2012 ) . Here we review previous attempts to improve upon this greedy growth strategy in the DT literature . Decision jungles ( Shotton et al. , 2013 ) employ a training mechanism to merge partitioned input spaces between different sub-trees , and thus to rectify suboptimal “ splits ” made due to the locality of optimisation . İrsoy et al . ( 2014 ) proposes budding trees , which are grown and pruned incrementally based on global optimisation of all existing nodes . While our proposed training algorithm , for simplicity , grows the architecture by greedily choosing the best option between going deeper and splitting the input space ( see Fig . 1 ) , it is certainly amenable to the above advances . Another related strand of work for feature learning is cascaded forests—stacks of RFs where the outputs of intermediate models are fed into the subsequent ones ( Montillo et al. , 2011 ; Kontschieder et al. , 2013 ; Zhou & Feng , 2017 ) . It has been shown how a cascade of DTs can be mapped to NNs with sparse connections ( Sethi , 1990 ) , and more recently Richmond et al . ( 2015 ) extended this argument to RFs . However , the features obtained in this approach are the intermediate outputs of respective component models , which are not optimised for the target task , and can not be learned end-to-end , thus limiting its representational quality . 3 ADAPTIVE NEURAL TREES . We now formalise the definition of Adaptive Neural Trees ( ANTs ) , which are a form of DTs enhanced with deep , learned representations . We focus on supervised learning , where the aim is to learn the conditional distribution p ( y|x ) from a set of N labelled samples ( x ( 1 ) , y ( 1 ) ) , ... , ( x ( N ) , y ( N ) ) ∈ X × Y as training data . 3.1 MODEL TOPOLOGY AND OPERATIONS . In short , an ANT is a tree-structured model , characterized by a set of hierarchical partitions of the input spaceX , a series of nonlinear transformations , and separate predictive models in the respective component regions . More formally , we define an ANT as a pair ( T , O ) where T defines the model topology , and O denotes the set of operations on it . We restrict the model topology T to be instances of binary trees , defined as a set of finite graphs where every node is either an internal node or a leaf , and is the child of exactly one parent node ( apart from the parent-less root node ) . We define the topology of a tree as T : = { N , E } where N is the set of all nodes , and E is the set of edges between them . Nodes with no children are leaf nodes , Nleaf , and all others are internal nodes , Nint . Every internal node j ∈ Nint has exactly two children nodes , represented by left ( j ) and right ( j ) . Unlike standard trees , E contains an edge which connects input data x with the root node , as shown in Fig.1 ( Left ) . Every node and edge is assigned with operations which acts on the allocated samples of data ( Fig.1 ) . Starting at the root , each sample gets transformed and traverses the tree according to the set of operations O . An ANT is constructed based on three primitive modules of differentiable operations : 1 . Routers , R : each internal node j ∈ Nint holds a router module , rθj : Xj → [ 0 , 1 ] ∈ R , parametrised by θ , which sends samples from the incoming edge to either the left or right child . Here Xj denotes the representation at node j . We use stochastic routing , where the binary decision ( 1 for the left and 0 for the right branch ) is sampled from Bernoulli distribution with mean rθj ( xj ) for input xj ∈ Xj . As an example , rθj can be defined as a small convolutional neural network ( CNN ) . 2 . Transformers , T : every edge e ∈ E of the tree has one or a composition of multiple transformer module ( s ) . Each transformer tψe ∈ T is a nonlinear function , parametrised by ψ , that transforms samples from the previous module and passes them to the next one . For example , tψe can be a single convolutional layer followed by ReLU ( Nair & Hinton , 2010 ) . Unlike in standard DTs , edges transform data and are allowed to “ grow ” by adding more operations ( Sec . 4 ) , learning “ deeper ” representations as needed . 3 . Solvers , S : each leaf node l ∈ Nleaf is assigned to a solver module , sφl : Xl → Y ∈ S , parametrised by φ , which operates on the transformed input data and outputs an estimate for the conditional distribution p ( y|x ) . For classification tasks , we can define , for example , sφ as a linear classifier on the feature space Xl , which outputs a distribution over classes . Defining operations on the graph T amounts to a specification of the triplet O = ( R , T , S ) . For example , given image inputs , we would choose the operations of each module to be from the set of operations commonly used in CNNs ( examples are given in Tab . 2 ) . In this case , every computational path on the resultant ANT , as well as the set of routers that guide inputs to one of these paths , are given by CNNs . In Sec . 4 , we discuss methods for constructing such tree-shaped NNs end-toend from simple building blocks . Lastly , many existing tree-structured models ( Suárez & Lutsko , 1999 ; İrsoy et al. , 2012 ; Laptev & Buhmann , 2014 ; Rota Bulo & Kontschieder , 2014 ; Kontschieder et al. , 2015 ; Frosst & Hinton , 2017 ; Xiao , 2017 ) are instantiations of ANTs with limitations which we will address with our model ( see Sec . 2 for a more detailed discussion ) .

Summary: The authors proposed a new model Adaptive Neural Trees(ANTs) by combining the representation learning and gradient optimization of neural networks with architecture learning of decision trees. The key advantage of the new model ANTs  over the existing methods(Random forest, Linear classifier, Neural decision forest, et al) is: it may achieve high accuracy(above $90\%$) with relatively much smaller number of parameters, as shown by the experiments on the datasets MNIST and CIFAR-10. Besides, the authors proposed single-path inference based on the greedily-selected leaf node to approximate the multi-path inferences with the full predictive distribution. The experiments show the single-path inference doesn't lose much accuracy but it saves memory and time. This paper is acceptable after minor modification.