Tree in Tree: from Decision Trees to Decision Graphs

1 Introduction . Decision trees ( DTs ) and tree ensembles are widely used in practice , particularly for applications that require few parameters [ 1–5 ] , fast inference [ 6–8 ] , and good interpretability [ 9 , 10 ] . In a DT , the internal and leaf nodes are organized in a binary structure , with internal nodes defining the routing function and leaf nodes predicting the class label . Although DTs are easy to train by recursively splitting leaf nodes , the tree structure can be suboptimal for the following reasons : ( 1 ) DTs can grow exponentially large as the depth of the tree increases . Yet , the root-leaf path can be short even for large DTs , limiting the predictive power . ( 2 ) In a DT , the nodes are not shared across different paths , reducing the efficiency of the model . Decision trees are similar to neural networks ( NNs ) in that both models are composed of basic units . A possible way to enhance the performance of DTs or NNs is to replace the basic units with more powerful models . For instance , “ Network in Network ” builds micro NNs with complex structures within local receptive fields to achieve state-of-the-art performances on image recognition tasks [ 11 ] . As for DTs , previous work replaced the axis-aligned splits with logistic regression or linear support vector machines to construct oblique trees [ 1 , 3 , 6 , 12–14 ] . The work in [ 5 ] further incorporates convolution operations into DTs for improved performance on image recognition tasks , while [ 1 ] replaces the leaf predictors with linear regression to improve the regression performance . Unlike the greedy training algorithms used for axis-aligned trees ( e.g. , Classification and Regression Trees or CART [ 15 ] ) , oblique trees are generally trained by gradient-based [ 3 , 13 , 14 ] or alternating [ 1 , 6 ] optimization algorithms . Inspired by the concepts of Network in Network [ 11 ] and oblique trees [ 6 , 12 ] , we propose a novel model , Tree in Tree ( TnT ) , to recursively replace the internal and leaf nodes with micro decision trees . In contrast to a conventional tree structure , the nodes in a TnT form a Directed Acyclic Graph ( DAG ) to address the aforementioned limitations and construct a more efficient model . Unlike previous oblique trees that were optimized on a predefined tree structure [ 1 , 5 ] , TnT can learn graph 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . connections from scratch . The major contributions of this work are as follows : ( 1 ) We extend decision trees to decision graphs and propose a scalable algorithm to construct large decision graphs . ( 2 ) We show that the proposed algorithm outperforms existing decision trees/graphs , either as a stand-alone classifier or base estimator in an ensemble , under the same model complexity constraints . ( 3 ) Rather than relying on a predefined graph/tree structure , the proposed algorithm is capable of learning graph connections from scratch ( i.e. , starting from a single leaf node ) and offers a fully interpretable decision process . We provide a Python implementation of the proposed TnT decision graph at https : //github.com/BingzhaoZhu/TnTDecisionGraph . 2 Related work Algorithm 1 : Naive decision graph ( NDG ) [ 16 ] 1 G← initialize graph with a leaf node ; 2 for i← 1 to N do 3 for each leaf node ( li ) ∈ G do 4 Find the maximum gain ( gi ) if we split li ; 5 for each pair of leaf nodes ( li , lj ) ∈ G do 6 Record gain ( gi , j ) if we merge li and lj ; 7 Split/merge nodes to maximize gain ; 8 Note : The split operation has a model complexity penalty ( C ) for creating an internal node . Decision graph ( DG ) is a generalization of the conventional decision tree algorithm , extending the tree structure to a directed acyclic graph [ 16 , 17 ] . Despite similarity in using a sequential inference scheme , training and optimizing DGs is more challenging due to the large search space for the graph structure . The work in [ 16 ] and [ 18 ] proposed a greedy algorithm to train DGs by tentatively joining pairs of leaf nodes at each training step ( NDG , Algorithm 1 ) . [ 19 ] constructed the DG as a Markov decision process , where base estimators were cascaded in the form of a datadependent AdaBoost ensemble . [ 20 ] combined multiple binary classifiers to construct a DAG for efficient multi-class classification . While [ 19 ] and [ 20 ] constructed DAGs with specific settings ( e.g. , Adaboost [ 19 ] and multi-class classification [ 20 ] ) , these methods do not provide a general extension of decision trees . [ 21 ] improved NDG by jointly optimizing split nodes and their connections to next-layer nodes . However , simultaneously learning the split and branching structure has no exact solution and relies on search-based algorithms to reach local minimum . Alternatively , in this work , we revisit the concept of decision graphs by exploiting recent advances in non-greedy tree optimization algorithms [ 6 , 12 , 14 , 22 ] . Our proposed Tree in Tree algorithm can construct DGs as a more accurate and efficient alternative to the widely-used decision trees , both as stand-alone classifiers and as weak learners in the ensembles . Conventional decision tree learning algorithms such as CART [ 15 ] and its variations follow a greedy top-down growing scheme . Recent work has focused on optimizing the structure of the tree [ 22–24 ] . However , constructing an optimal binary DT is NP-hard [ 25 ] and optimal trees are not scalable to large datasets with many samples and features [ 22–24 ] . Recent studies have further developed scalable algorithms for non-greedy decision tree optimization , with no guarantee on tree optimality [ 1 , 3 , 6 , 12–14 ] . Such scalable approaches can be categorized into two groups : tree alternating optimization ( TAO ) [ 1 , 6 ] and gradient-based optimization [ 3 , 12–14 ] . TAO decomposes the tree optimization problem into a set of reduced problems imposed at the node levels . The work in [ 6 ] applied the alternating optimization to both axis-aligned trees and sparse oblique trees . Later , [ 1 ] extended TAO to regression tasks and ensemble methods . Unlike TAO , gradient-based optimization requires a differentiable objective function , which can be obtained by different methods . For example , [ 12 ] derived a convex-concave upper bound of the empirical loss . [ 13 ] and [ 3 ] considered a soft ( i.e. , probabilistic ) split at the internal nodes and formulated a global objective function . The activation function for soft splits was refined in [ 14 ] to enable conditional inference and parameter update . Both TAO and gradient-based optimization operate on a predefined tree structure and optimize the parameters of the internal nodes . The proposed Tree in Tree algorithm aims to optimize the graph/tree structure by growing micro decision trees inside current nodes . Compared to the greedy top-down tree induction [ 15 ] , Tree in Tree solves a reduced optimization problem at each node , which is enabled via non-greedy tree alternating optimization techniques [ 6 ] . Compared to NDG , TnT employs a non-greedy process to construct decision graphs , which leads to an improved classification performance ( discussed in later sections ) . Compared to axis-aligned decision trees ( e.g. , TAO [ 1 , 6 ] , CART [ 15 ] ) , TnT extends the tree structure to a more accurate and compact directed acyclic graph , in which nodes are shared across multiple paths . 3 Methods . In this work , we consider a classification task with input and output spaces denoted by X ⊂ RD and Y = { 1 , ... , K } , respectively . Similar to conventional decision trees , a decision graph classifier G consists of internal nodes and leaf nodes . Each internal node is assigned a binary split function s ( · ; θ ) : X → [ left_child , right_child ] parametrized by θ , which defines the routing function of a graph . For axis-aligned splits , θ indicates a feature index and a threshold . The terminal nodes ( with no children ) are named leaf nodes and indicate the class labels . 3.1 Decision graph . As an extension to the tree structure , decision graphs organize the nodes into a more generic directed acyclic graph . In this work , we limit our discussion to axis-aligned binary DTs/DGs in which each internal node compares a feature value to a threshold to select one of the two child nodes . Similar to the sequential inference process in DTs , the test samples in a DG start from the root and successively select a path at the internal nodes until a leaf node is reached . The main differences between binary DTs and DGs are the following : ( 1 ) In DTs , each child node has one parent node . However , DGs allow multiple parent nodes to share the same child node . Therefore , DG can combine the nodes with similar behaviors ( e.g. , similar split functions ) to reduce model complexity . ( 2 ) In binary DTs , the number of leaf nodes is always greater than the internal nodes by one . In DGs , however , # Leaves ≤ # Internals+ 1 , since multiple internal nodes can share the same leaf node . Furthermore , there exists a unique path to reach each leaf node in a tree structure , which does not hold within DGs . ( 3 ) The model complexity of a DT is often quantified by the number of internal or leaf nodes . However , we can post-process a DG by merging the leaf nodes with the same class label . As a result , DGs have a minimum leaf node count equal to the number of classes . Therefore , we use the number of splits ( i.e. , internal nodes ) to quantify the model complexity of a DG . 3.2 Tree in Tree We propose a novel algorithm named Tree in Tree as a scalable method to construct large decision graphs . Conventional DT training algorithms ( e.g. , CART ) are greedy and recursively split the leaf nodes to grow a deep structure , without optimizing the previously learned split functions . The key difference between the proposed TnT model and conventional approaches lies in the optimization of the internal nodes . TnT fits new decision trees in place of the internal/leaf nodes and employs such micro DTs to construct a directed acyclic graph . Overall , the proposed TnT model is a novel extension to the conventional decision trees and generates accurate predictions by routing samples through a directed acyclic graph . Figure 1 shows the high-level procedure for training a decision graph with the proposed TnT algorithm . Assuming a starting decision graph ( e.g. , a decision tree or a single leaf node ) , our goal is to grow a larger model with improved predictive power . In the growing phase of TnT ( Fig . 1 ( a ) ) , we replace a node ( dashed circle ) with a micro decision tree with multiple splits to enable more accurate decision boundaries . In the merging phase ( Fig . 1 ( b ) ) , the micro decision tree is merged into the starting model to construct a TnT decision graph , in which a child node ( node 2 ) may have multiple parent nodes ( node 4 and 5 ) . Growing the graph from internal nodes We consider the training of a decision graph as an optimization problem with the aim of minimizing the loss function on the training data : min ∑ x , y∈X , Y L ( y , G ( x ; Θ ) ) . ( 1 ) TnT grows the decision graph G ( · ; Θ ) from an arbitrary internal node ni ∈ G with the split function s ( · ; θi ) . θi denotes the trainable parameters of ni including a feature index and a threshold for axis-aligned splits . The overall goal is to replace ni with a decision tree ti and minimize the loss function as indicated in ( 1 ) . All other nodes remain unchanged as we train ti . Let us consider a subset of samples ( xsubset ∈ Xsubset , ysubset ∈ Ysubset ) that is sensitive to the split function s ( · ; θi ) , as defined by the following expression : Gni→left ( xsubset ; Θ\θi ) 6= Gni→right ( xsubset ; Θ\θi ) , ( 2 ) where Θ\θi denotes the parameters of all nodes in G excluding ni . Growing the graph from ni does not change Θ\θi since all other nodes are fixed as we solve the reduced optimization problem at ni . Gni→left sends the samples to the left child at ni ( i.e. , s ( · ; θi ) → left_child ) while Gni→right routes the samples to the right child at ni . With Θ\θi being fixed , the output of decision graph only depends on θi ( i.e. , s ( · ; θi ) ) G ( x ; Θ ) = { Gni→left ( x ; Θ\θi ) if s ( x ; θi ) → left_child Gni→right ( x ; Θ\θi ) if s ( x ; θi ) → right_child . ( 3 ) Since L ( y , Gni→left ( x ; Θ\θi ) ) 6= L ( y , Gni→right ( x ; Θ\θi ) ) only if the inequality ( 2 ) holds , we can solve the reduced optimization problem at node ni based on the subset ( Xsubset , Ysubset ) instead of using the entire training set : min θi ∑ x∈Xsubset y∈Ysubset L ( y , G ( x ; Θ ) ) . ( 4 ) Having Equation ( 3 ) , the optimization problem ( 4 ) has a closed-form solution as follows : t∗i ( x ) : = { left_child if L ( y , Gni→left ( x ; Θ\θi ) ) < L ( y , Gni→right ( x ; Θ\θi ) ) right_child if L ( y , Gni→right ( x ; Θ\θi ) ) < L ( y , Gni→left ( x ; Θ\θi ) ) . ( 5 ) Equation ( 5 ) defines the optimal split function at the internal node ni which is used to fit the micro decision tree ti . With other nodes being fixed , we show that the overall loss function of G can be minimized by pursuing the optimal spilt function at an arbitrary internal node ni . Rather than using a simple axis-aligned split , the proposed TnT algorithm learns a complexity-constrained decision tree to better approximate the optimal split function ( Equation ( 5 ) ) . Growing the graph from leaf nodes Growing from the leaf nodes is a standard practice in greedy training algorithms , where we recursively split the leaf nodes to achieve a deeper tree with a better fit on the training data [ 15 ] . In TnT , we replace the leaf predictors with decision trees . Let G ( · ; Θ ) be a decision graph and nl ∈ G an arbitrary leaf node with a constant class label l ( · ; θl ) = c. Our goal is to minimize the overall loss function L ( Y , G ( X ; Θ ) ) by replacing the leaf predictor l ( · ; θl ) with a micro decision tree tl ( x ) . Consider the subset of samples ( Xsubset , Ysubset ) that visit the leaf node nl . Minimization of the loss function ( 1 ) can be expressed as min θl ∑ x∈Xsubset y∈Ysubset L ( y , tl ( x ) ) , ( 6 ) where the minimum is simply achieved at t∗l ( x ) : = y for x , y ∈ Xsubset , Ysubset ( i.e. , the ideal leaf predictor ) . We build a decision tree to approximate the ideal leaf predictor .

Summary: The authors present a new algorithm for large decision graphs. Tree in a tree is a scalable method that fits new decision trees in place of the internal/leaf nodes and constructs a directed acyclic graph. The authors show that their method outperforms other learning models in different datasets with different configurations.