user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adaptive Neural Trees

Paper content:
Deep neural networks and decision trees operate on largely separate paradigms; typically, the former performs representation learning with pre-specified architectures, while the latter is characterised by learning hierarchies over pre-specified features with data-driven architectures.
We unite the two via adaptive neural trees (ANTs), a model that incorporates representation learning into edges, routing functions and leaf nodes of a decision tree, along with a backpropagation-based training algorithm that adaptively grows the architecture from primitive modules (e.g., convolutional layers).
ANTs allow increased interpretability via hierarchical clustering, e.g., learning meaningful class associations, such as separating natural vs. man-made objects.
We demonstrate this on classification and regression tasks, achieving over 99% and 90% accuracy on the MNIST and CIFAR-10 datasets, and outperforming standard neural networks, random forests and gradient boosted trees on the SARCOS dataset.
Furthermore, ANT optimisation naturally adapts the architecture to the size and complexity of the training data.
Neural networks (NNs) and decision trees (DTs) are both powerful classes of machine learning models with proven successes in academic and commercial applications.
The two approaches, however, typically come with mutually exclusive benefits and limitations.NNs are characterised by learning hierarchical representations of data through the composition of nonlinear transformations BID59 BID2 , which has alleviated the need for feature engineering, in contrast with many other machine learning models.
In addition, NNs are trained with stochastic optimisers, such as stochastic gradient descent (SGD), allowing training to scale to large datasets.
Consequently, with modern hardware, we can train NNs of many layers on large datasets, solving numerous problems ranging from object detection to speech recognition with unprecedented accuracy BID35 .
However, their architectures typically need to be designed by hand and fixed per task or dataset, requiring domain expertise BID62 .
Inference can also be heavy-weight for large models, as each sample engages every part of the network, i.e., increasing capacity causes a proportional increase in computation .Alternatively
, DTs are characterised by learning hierarchical clusters of data . A DT learns
how to split the input space, so that in each subset, linear models suffice to explain the data. In contrast
to standard NNs, the architectures of DTs are optimised based on training data, and are particularly advantageous in data-scarce scenarios. DTs also enjoy
lightweight inference as only a single root-to-leaf path on the tree is used for each input sample. However, successful
applications of DTs often require hand-engineered features of data. We can ascribe the
limited expressivity of single DTs to the common use of simplistic routing functions, such as splitting on axis-aligned features. The loss function
for optimising hard partitioning is non-differentiable, which hinders the use of gradient descent-based optimization and thus complex splitting functions. Current techniques
for increasing capacity include ensemble methods such as random forests (RFs) BID4 ) and gradient-boosted trees (GBTs) BID14 , which are known to achieve state-of-the-art performance in various tasks, including medical imaging and financial forecasting BID49 BID27 BID33 BID56 .The goal of this work
is to combine NNs and DTs to gain the complementary benefits of both approaches. To this end, we propose
adaptive neural trees (ANTs), which generalise previous work that attempted the same unification (Suárez & Lutsko, 1999; İrsoy et al., 2012; BID32 BID46 BID30 BID57 and address their limitations (see Tab. 1). ANTs represent routing
decisions and root-to-leaf computational paths within the tree structures as NNs, which lets them benefit from hierarchical representation learning, rather than being restricted to partitioning the raw data space. In addition, we propose
a backpropagation-based training algorithm to grow ANTs based on a series of decisions between making the ANT deeper-the central NN paradigm-or partitioning the data-the central DT paradigm (see FIG0 ). This allows the architectures
of ANTs to adapt to the data available. By our design, ANTs inherit the
following desirable properties from both DTs and NNs:• Representation learning: as each root-to-leaf path in an ANT is a NN, features can be learnt end-to-end with gradient-based optimisation. This, in turn, allows for learning
complex data partitioning. The training algorithm is also amenable
to SGD. • Architecture learning: by progressively
growing ANTs, the architecture adapts to the availability and complexity of data, embodying Occams razor. The growth procedure can be viewed as architecture
search with a hard constraint over the model class.• Lightweight inference: at inference time, ANTs perform
conditional computation, selecting a single root-to-leaf path on the tree on a per-sample basis, activating only a subset of the parameters of the model.We empirically validate these benefits for classification and regression through experiments on the MNIST BID34 , CIFAR-10 (Krizhevsky & Hinton, 2009 ) and SARCOS BID55 datasets. Along with other forms of neural networks, ANTs far outperform
state-of-the-art random forest (RF) BID61 and gradient boosted tree (GBT) BID43 ) methods on the image-based classification datasets, with architectures achieving over 99% accuracy on MNIST and over 90% accuracy on CIFAR-10. On the other hand, the best performing methods on the SARCOS multivariate
regression dataset are all tree-based, with soft decision trees (SDTs) (Suárez & Lutsko, 1999; BID26 , GBTs (Friedman, 2001) and ANTs achieving the lowest mean squared error. At the same time, ANTs can learn meaningful hierarchical partitionings of
data, e.g., grouping man-made and natural objects (see FIG2 ). ANTs also have reduced time and memory requirements during inference, conferred
by conditional computation. In one case, we discover an architecture that achieves over 98% accuracy on MNIST
using approximately the same number of parameters as a linear classifier on raw image pixels, showing the benefits of modelling a hierarchical structure that reflects the underlying data structure in enhancing both computational and predictive performance. Finally, we demonstrate the benefits of architecture learning by training ANTs on
subsets of CIFAR-10 of varying sizes. The method can construct architectures of adequate size, leading to better generalisation
, particularly on small datasets.
We introduced Adaptive Neural Trees (ANTs), a holistic way to marry the architecture learning, conditional computation and hierarchical clustering of decision trees (DTs) with the hierarchical representation learning and gradient descent optimization of deep neural networks (DNNs).
Our proposed training algorithm optimises both the parameters and architectures of ANTs through progressive growth, tuning them to the size and complexity of the training dataset.
Together, these properties make ANTs a generalisation of previous work attempting to unite NNs and DTs.
Finally, we validated the claimed benefits of ANTs on standard regression and object classification datasets, whilst still achieving high performance.

assistant: text: We propose a framework to combine decision trees and neural networks, and show on image classification tasks that it enjoys the complementary benefits of the two approaches, while addressing the limitations of prior work.