Tree in Tree: from Decision Trees to Decision Graphs

1 Introduction . Decision trees ( DTs ) and tree ensembles are widely used in practice , particularly for applications that require few parameters [ 1–5 ] , fast inference [ 6–8 ] , and good interpretability [ 9 , 10 ] . In a DT , the internal and leaf nodes are organized in a binary structure , with internal nodes defining the routing function and leaf nodes predicting the class label . Although DTs are easy to train by recursively splitting leaf nodes , the tree structure can be suboptimal for the following reasons : ( 1 ) DTs can grow exponentially large as the depth of the tree increases . Yet , the root-leaf path can be short even for large DTs , limiting the predictive power . ( 2 ) In a DT , the nodes are not shared across different paths , reducing the efficiency of the model . Decision trees are similar to neural networks ( NNs ) in that both models are composed of basic units . A possible way to enhance the performance of DTs or NNs is to replace the basic units with more powerful models . For instance , “ Network in Network ” builds micro NNs with complex structures within local receptive fields to achieve state-of-the-art performances on image recognition tasks [ 11 ] . As for DTs , previous work replaced the axis-aligned splits with logistic regression or linear support vector machines to construct oblique trees [ 1 , 3 , 6 , 12–14 ] . The work in [ 5 ] further incorporates convolution operations into DTs for improved performance on image recognition tasks , while [ 1 ] replaces the leaf predictors with linear regression to improve the regression performance . Unlike the greedy training algorithms used for axis-aligned trees ( e.g. , Classification and Regression Trees or CART [ 15 ] ) , oblique trees are generally trained by gradient-based [ 3 , 13 , 14 ] or alternating [ 1 , 6 ] optimization algorithms . Inspired by the concepts of Network in Network [ 11 ] and oblique trees [ 6 , 12 ] , we propose a novel model , Tree in Tree ( TnT ) , to recursively replace the internal and leaf nodes with micro decision trees . In contrast to a conventional tree structure , the nodes in a TnT form a Directed Acyclic Graph ( DAG ) to address the aforementioned limitations and construct a more efficient model . Unlike previous oblique trees that were optimized on a predefined tree structure [ 1 , 5 ] , TnT can learn graph 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . connections from scratch . The major contributions of this work are as follows : ( 1 ) We extend decision trees to decision graphs and propose a scalable algorithm to construct large decision graphs . ( 2 ) We show that the proposed algorithm outperforms existing decision trees/graphs , either as a stand-alone classifier or base estimator in an ensemble , under the same model complexity constraints . ( 3 ) Rather than relying on a predefined graph/tree structure , the proposed algorithm is capable of learning graph connections from scratch ( i.e. , starting from a single leaf node ) and offers a fully interpretable decision process . We provide a Python implementation of the proposed TnT decision graph at https : //github.com/BingzhaoZhu/TnTDecisionGraph . 2 Related work Algorithm 1 : Naive decision graph ( NDG ) [ 16 ] 1 G← initialize graph with a leaf node ; 2 for i← 1 to N do 3 for each leaf node ( li ) ∈ G do 4 Find the maximum gain ( gi ) if we split li ; 5 for each pair of leaf nodes ( li , lj ) ∈ G do 6 Record gain ( gi , j ) if we merge li and lj ; 7 Split/merge nodes to maximize gain ; 8 Note : The split operation has a model complexity penalty ( C ) for creating an internal node . Decision graph ( DG ) is a generalization of the conventional decision tree algorithm , extending the tree structure to a directed acyclic graph [ 16 , 17 ] . Despite similarity in using a sequential inference scheme , training and optimizing DGs is more challenging due to the large search space for the graph structure . The work in [ 16 ] and [ 18 ] proposed a greedy algorithm to train DGs by tentatively joining pairs of leaf nodes at each training step ( NDG , Algorithm 1 ) . [ 19 ] constructed the DG as a Markov decision process , where base estimators were cascaded in the form of a datadependent AdaBoost ensemble . [ 20 ] combined multiple binary classifiers to construct a DAG for efficient multi-class classification . While [ 19 ] and [ 20 ] constructed DAGs with specific settings ( e.g. , Adaboost [ 19 ] and multi-class classification [ 20 ] ) , these methods do not provide a general extension of decision trees . [ 21 ] improved NDG by jointly optimizing split nodes and their connections to next-layer nodes . However , simultaneously learning the split and branching structure has no exact solution and relies on search-based algorithms to reach local minimum . Alternatively , in this work , we revisit the concept of decision graphs by exploiting recent advances in non-greedy tree optimization algorithms [ 6 , 12 , 14 , 22 ] . Our proposed Tree in Tree algorithm can construct DGs as a more accurate and efficient alternative to the widely-used decision trees , both as stand-alone classifiers and as weak learners in the ensembles . Conventional decision tree learning algorithms such as CART [ 15 ] and its variations follow a greedy top-down growing scheme . Recent work has focused on optimizing the structure of the tree [ 22–24 ] . However , constructing an optimal binary DT is NP-hard [ 25 ] and optimal trees are not scalable to large datasets with many samples and features [ 22–24 ] . Recent studies have further developed scalable algorithms for non-greedy decision tree optimization , with no guarantee on tree optimality [ 1 , 3 , 6 , 12–14 ] . Such scalable approaches can be categorized into two groups : tree alternating optimization ( TAO ) [ 1 , 6 ] and gradient-based optimization [ 3 , 12–14 ] . TAO decomposes the tree optimization problem into a set of reduced problems imposed at the node levels . The work in [ 6 ] applied the alternating optimization to both axis-aligned trees and sparse oblique trees . Later , [ 1 ] extended TAO to regression tasks and ensemble methods . Unlike TAO , gradient-based optimization requires a differentiable objective function , which can be obtained by different methods . For example , [ 12 ] derived a convex-concave upper bound of the empirical loss . [ 13 ] and [ 3 ] considered a soft ( i.e. , probabilistic ) split at the internal nodes and formulated a global objective function . The activation function for soft splits was refined in [ 14 ] to enable conditional inference and parameter update . Both TAO and gradient-based optimization operate on a predefined tree structure and optimize the parameters of the internal nodes . The proposed Tree in Tree algorithm aims to optimize the graph/tree structure by growing micro decision trees inside current nodes . Compared to the greedy top-down tree induction [ 15 ] , Tree in Tree solves a reduced optimization problem at each node , which is enabled via non-greedy tree alternating optimization techniques [ 6 ] . Compared to NDG , TnT employs a non-greedy process to construct decision graphs , which leads to an improved classification performance ( discussed in later sections ) . Compared to axis-aligned decision trees ( e.g. , TAO [ 1 , 6 ] , CART [ 15 ] ) , TnT extends the tree structure to a more accurate and compact directed acyclic graph , in which nodes are shared across multiple paths . 3 Methods . In this work , we consider a classification task with input and output spaces denoted by X ⊂ RD and Y = { 1 , ... , K } , respectively . Similar to conventional decision trees , a decision graph classifier G consists of internal nodes and leaf nodes . Each internal node is assigned a binary split function s ( · ; θ ) : X → [ left_child , right_child ] parametrized by θ , which defines the routing function of a graph . For axis-aligned splits , θ indicates a feature index and a threshold . The terminal nodes ( with no children ) are named leaf nodes and indicate the class labels . 3.1 Decision graph . As an extension to the tree structure , decision graphs organize the nodes into a more generic directed acyclic graph . In this work , we limit our discussion to axis-aligned binary DTs/DGs in which each internal node compares a feature value to a threshold to select one of the two child nodes . Similar to the sequential inference process in DTs , the test samples in a DG start from the root and successively select a path at the internal nodes until a leaf node is reached . The main differences between binary DTs and DGs are the following : ( 1 ) In DTs , each child node has one parent node . However , DGs allow multiple parent nodes to share the same child node . Therefore , DG can combine the nodes with similar behaviors ( e.g. , similar split functions ) to reduce model complexity . ( 2 ) In binary DTs , the number of leaf nodes is always greater than the internal nodes by one . In DGs , however , # Leaves ≤ # Internals+ 1 , since multiple internal nodes can share the same leaf node . Furthermore , there exists a unique path to reach each leaf node in a tree structure , which does not hold within DGs . ( 3 ) The model complexity of a DT is often quantified by the number of internal or leaf nodes . However , we can post-process a DG by merging the leaf nodes with the same class label . As a result , DGs have a minimum leaf node count equal to the number of classes . Therefore , we use the number of splits ( i.e. , internal nodes ) to quantify the model complexity of a DG . 3.2 Tree in Tree We propose a novel algorithm named Tree in Tree as a scalable method to construct large decision graphs . Conventional DT training algorithms ( e.g. , CART ) are greedy and recursively split the leaf nodes to grow a deep structure , without optimizing the previously learned split functions . The key difference between the proposed TnT model and conventional approaches lies in the optimization of the internal nodes . TnT fits new decision trees in place of the internal/leaf nodes and employs such micro DTs to construct a directed acyclic graph . Overall , the proposed TnT model is a novel extension to the conventional decision trees and generates accurate predictions by routing samples through a directed acyclic graph . Figure 1 shows the high-level procedure for training a decision graph with the proposed TnT algorithm . Assuming a starting decision graph ( e.g. , a decision tree or a single leaf node ) , our goal is to grow a larger model with improved predictive power . In the growing phase of TnT ( Fig . 1 ( a ) ) , we replace a node ( dashed circle ) with a micro decision tree with multiple splits to enable more accurate decision boundaries . In the merging phase ( Fig . 1 ( b ) ) , the micro decision tree is merged into the starting model to construct a TnT decision graph , in which a child node ( node 2 ) may have multiple parent nodes ( node 4 and 5 ) . Growing the graph from internal nodes We consider the training of a decision graph as an optimization problem with the aim of minimizing the loss function on the training data : min ∑ x , y∈X , Y L ( y , G ( x ; Θ ) ) . ( 1 ) TnT grows the decision graph G ( · ; Θ ) from an arbitrary internal node ni ∈ G with the split function s ( · ; θi ) . θi denotes the trainable parameters of ni including a feature index and a threshold for axis-aligned splits . The overall goal is to replace ni with a decision tree ti and minimize the loss function as indicated in ( 1 ) . All other nodes remain unchanged as we train ti . Let us consider a subset of samples ( xsubset ∈ Xsubset , ysubset ∈ Ysubset ) that is sensitive to the split function s ( · ; θi ) , as defined by the following expression : Gni→left ( xsubset ; Θ\θi ) 6= Gni→right ( xsubset ; Θ\θi ) , ( 2 ) where Θ\θi denotes the parameters of all nodes in G excluding ni . Growing the graph from ni does not change Θ\θi since all other nodes are fixed as we solve the reduced optimization problem at ni . Gni→left sends the samples to the left child at ni ( i.e. , s ( · ; θi ) → left_child ) while Gni→right routes the samples to the right child at ni . With Θ\θi being fixed , the output of decision graph only depends on θi ( i.e. , s ( · ; θi ) ) G ( x ; Θ ) = { Gni→left ( x ; Θ\θi ) if s ( x ; θi ) → left_child Gni→right ( x ; Θ\θi ) if s ( x ; θi ) → right_child . ( 3 ) Since L ( y , Gni→left ( x ; Θ\θi ) ) 6= L ( y , Gni→right ( x ; Θ\θi ) ) only if the inequality ( 2 ) holds , we can solve the reduced optimization problem at node ni based on the subset ( Xsubset , Ysubset ) instead of using the entire training set : min θi ∑ x∈Xsubset y∈Ysubset L ( y , G ( x ; Θ ) ) . ( 4 ) Having Equation ( 3 ) , the optimization problem ( 4 ) has a closed-form solution as follows : t∗i ( x ) : = { left_child if L ( y , Gni→left ( x ; Θ\θi ) ) < L ( y , Gni→right ( x ; Θ\θi ) ) right_child if L ( y , Gni→right ( x ; Θ\θi ) ) < L ( y , Gni→left ( x ; Θ\θi ) ) . ( 5 ) Equation ( 5 ) defines the optimal split function at the internal node ni which is used to fit the micro decision tree ti . With other nodes being fixed , we show that the overall loss function of G can be minimized by pursuing the optimal spilt function at an arbitrary internal node ni . Rather than using a simple axis-aligned split , the proposed TnT algorithm learns a complexity-constrained decision tree to better approximate the optimal split function ( Equation ( 5 ) ) . Growing the graph from leaf nodes Growing from the leaf nodes is a standard practice in greedy training algorithms , where we recursively split the leaf nodes to achieve a deeper tree with a better fit on the training data [ 15 ] . In TnT , we replace the leaf predictors with decision trees . Let G ( · ; Θ ) be a decision graph and nl ∈ G an arbitrary leaf node with a constant class label l ( · ; θl ) = c. Our goal is to minimize the overall loss function L ( Y , G ( X ; Θ ) ) by replacing the leaf predictor l ( · ; θl ) with a micro decision tree tl ( x ) . Consider the subset of samples ( Xsubset , Ysubset ) that visit the leaf node nl . Minimization of the loss function ( 1 ) can be expressed as min θl ∑ x∈Xsubset y∈Ysubset L ( y , tl ( x ) ) , ( 6 ) where the minimum is simply achieved at t∗l ( x ) : = y for x , y ∈ Xsubset , Ysubset ( i.e. , the ideal leaf predictor ) . We build a decision tree to approximate the ideal leaf predictor .

Summary: The paper proposes an algorithm to build decision acyclic graphs with properties similar to those of decision trees: each node has a split condition to route instances and terminal nodes of the graph (i.e. those without children) provide the classification labels.  This model is trained by creating embedded decision trees inside decision the graph nodes iteratively. The algorithm starts with a single leave node in which, a small decision tree is build to route the examples to left/right. Then the decision tree internal decision tree of the node is flattened to create a graph. The acyclic graph structure appears since all left/right decisions of the embedded tree are routed to the same left/right graph node. Then, for each node the same process is applied iteratively.  The method is used as the base classifier of bagging and Adaboost ensembles and compared on 8 datasets against standard decision trees with RF and Adaboost using the same over all complexity (complexity measured as number of splits).