Generalizing Tree Models for Improving Prediction Accuracy

1 INTRODUCTION . Can we generalize and improve the representation power of tree models ? The tree models learn a structure where the decision process is easy to follow ( Breiman et al. , 1984 ) . Due to this attractive feature of tree models , various efforts are being put to improve their performance or to utilize them as subcomponents of a deep learning model ( Kontschieder et al. , 2015 ; Shen et al. , 2018 ) . Compared to typical deep neural networks , the main characteristic of tree models is that input data are propagated through layers without a change in their representations ; the internal nodes calculate the probability of an input x arriving at the leaf nodes . The decision process that determines the membership of a training example , i.e. , the process of constructing the subsets of training data at each leaf , is the core operation in tree models , which we generalize and improve in this work . Previous works on tree models consider the decisions at different nodes as separate operations , i.e. , each node performs its own decision independently from the other nodes in the same layer , based on the arrival probability of an input x to the node . The independence of decisions allows learning to be simplified . However , in order to obtain a comprehensive view , the decisions of multiple nodes must be considered simultaneously . Furthermore , a typical tree model with a depth of L requires bL − 1 decision functions , where b is a branching factor , which makes it intractable to construct a deep tree model . In this work , we suggest that many tree models can be generalized to what we term as the decision transformer as Proposition 1 . A decision transformer generalizes existing tree models by treating the decisions of each layer as a single operation involving all nodes in that layer . More specifically , the decision transformer views each layer as a stochastic decision ( Def . 2 ) which linearly transforms the membership of each training data by a learned stochastic matrix ( Def . 1 ) . The aggregated layerwise view allows the decision transformer to reduce the complexity of analysis to O ( L ) , where L is the tree depth . Furthermore , formulating the tree models in a probabilistic context allows them to be understood by the deep learning framework and provides us a theoretical foundation for deeply understanding the advantages and the limitations of existing tree models in one framework . The in-depth understanding of tree models as decision transformers allows us to propose Decision Transformer Network ( DTN ) , a novel architecture that inherits the interpretability of tree models while improving their representation power . DTN is an extension of tree models into deep networks that creates multiple paths between nodes based on stochastic decisions . We propose two versions of DTN : DTN-D with full dense connections and DTN-S with sparse connections where the edges are connected in a locality-sensitive form . Our experiments show that DTN outperforms previous tree models and deep neural networks on 121 tabular datasets . Our contributions are summarized as follows . First , we introduce a generalized framework for tree models called the decision transformer and perform theoretical analysis on its generalizability and interpretability ( Section 3 ) . Second , we propose dense and sparse versions of Decision Transformer Network ( DTN ) , our novel decision-based model that is both interpretable and accurate ( Section 4 ) . Third , we undergo extensive experiments on 121 tabular datasets to show that DTN outperforms the state-of-the-art tree models and deep neural networks ( Section 5 ) . 2 RELATED WORKS . A tree model refers to a parameterized function that passes an example through its tree-structured layers without changing its representations . We categorize tree models based on whether it utilizes linear decisions over raw features or neural networks in combination with representation learning . Tree Models with Linear Decisions Traditional tree models perform linear decisions over a raw feature . Given an example x , decision trees ( DT ) ( Breiman et al. , 1984 ) compare a single selected element to a learned threshold at each branch to perform a decision . Soft decision trees ( SDT ) ( Irsoy et al. , 2012 ) generalize DTs to use all elements of x by a logistic classifier , making a probabilistic decision in each node . SDTs have been widely studied and applied to various problems due to its simplicity ( Irsoy & Alpaydin , 2015 ; Frosst & Hinton , 2017 ; Linero & Yang , 2018 ; Yoo & Sael , 2019 ) . Deep neural decision trees ( Yang et al. , 2018 ) also extend DTs to a multi-branched tree by splitting each feature element into multiple bins . These models provide a direct interpretation of predictions and decision processes . There are also well-known ensembles of such linear trees , such as random forests ( Breiman , 2001 ) and XGBoost ( Chen & Guestrin , 2016 ) . These ensemble models tboost the performance of tree models at the expense of interpretability . Tree Models with Representation Learning There are recent works combining tree models with deep neural networks , which add the ability of representation learning to the hierarchical decisions of tree models . A popular approach is to use abstract representations learned by convolutional neural networks ( CNNs ) as inputs of the decision functions of tree models ( Kontschieder et al. , 2015 ; Roy & Todorovic , 2016 ; Shen et al. , 2017 ; 2018 ; Wan et al. , 2020 ) , sometimes with multilayer perceptrons instead of CNNs ( Bulò & Kontschieder , 2014 ) . Another approach is to categorize raw examples by hierarchical decisions and then apply different classifiers to the separate clusters ( Murthy et al. , 2016 ) . There are also approaches that improve the performance of deep neural networks by inserting hierarchical decisions as differentiable operations into a neural network , instead of building a complete tree model ( Murdock et al. , 2016 ; McGill & Perona , 2017 ; Brust & Denzler , 2019 ; Tanno et al. , 2019 ) . In this work , we focus on models having complete trees in its structure . 3 GENERALIZING TREE MODELS . We first introduce a decision transformer , our unifying framework that represents a tree model as a series of linear transformations over probability vectors . We then analyze the properties of decision transformers : the generalizability to existing tree models and the interpretability . 3.1 DECISIONS TRANSFORMERS . Definition 1 ( Stochastic matrix ) . A matrix T is ( left ) stochastic if T ≥ 0 and ∑ i aij = 1 for all j . A vector p is stochastic if it satisfies the condition as a matrix of size |p| × 1 . We denote the set of all stochastic vectors and matrices as P , i.e. , we represent that T ∈ P or p ∈ P . Definition 2 ( Stochastic decision ) . A function f is a stochastic decision if it linearly transforms an input vector π by a left stochastic matrix as f ( π ) = Tπ , where T ∈ P is independent of π. Corollary 1 . Given a stochastic decision f and a vector π , f ( π ) ∈ P if π ∈ P . Based on the observation that each layer of a tree model can be represented as a stochastic decision that eventually learns the membership probability of an input x to the leaf nodes , we propose decision transformers which represent a tree model as a series of stochastic decisions whose transition matrices are determined by x . Proposition 1 ( Decision transformer ) . A parameterized classifierM is a decision transformer if it can be represented as M ( x ; Θ ) = FTL−1 · · ·T1T0π0 ∈ RC , ( 1 ) whereC is the number of target classes , π0 ∈ P is the initial decision probability , L is the number of decisions , Tl ∈ P is a transition matrix , and F ∈ P is a classifier matrix . Tl and F are calculated from parameterized functions Tl and F , respectively , that take x as inputs : Tl = Tl ( x ; θl ) ∈ RNl+1×Nl F = F ( x ; θL ) ∈ RC×NL , ( 2 ) whereNl is the size of decision vectors after the l-th transformation . The initial decision probability π0 is typically given as a hyperparameter , and thus Θ = { θ0 , · · · , θL } . We define Tl and F separately since internal and leaf nodes have different functionality in most tree models , although they are basically the same . For example , the lth internal layer of a typical decision tree updates a one-hot membership probability of x by a stochastic matrix Tl , which is generated by comparing an element of x to a learned threshold . In this case , Tl is a rectangular block-diagonal matrix having nonzero elements at ( 2j , j ) and ( 2j + 1 , j ) for every node j in the layer . On the other hand , the leaf layer utilizes a dense classification matrix F that makes the final prediction . Figure 1 illustrates two views of a binary tree model . Figure 1a treats a tree model as a set of independent decisions that are selected based on the path of x . On the other hand , Figure 1b represents the tree as a series of linear transformations , based on our framework of decision transformers . 3.2 GENERALIZING BINARY TREE MODELS . We now represent existing tree models as decision transformers and determine their characteristics . We work with binary trees for simplicity , but the analysis can be naturally extended to tree models having higher branching factors ( Yang et al. , 2018 ; Murthy et al. , 2016 ; Tanno et al. , 2019 ) . Binary tree models can be represented with block diagonal transition matrices with nonzero values on the 2 × 1 diagonal blocks , such that each column contains at most two nonzero elements that sum to one . Corollary 2 formalizes this notion . Corollary 2 . The l-th transition function Tl of a binary tree model is represented as Tl ( x ) = diag ( [ d1 ( x ) 1− d1 ( x ) ] , · · · , [ dn ( x ) 1− dn ( x ) ] ) ∈ R2 l+1×2l , ( 3 ) where diag ( · ) generates a rectangular block diagonal matrix , and di is the unit decision function of node i , which measures the probability of x for taking the left branch at node i . Based on Corollary 2 , we represent various tree models with specific forms of transition functions as summarized in Table 1 . The unit decision function of decision trees ( DT ) selects a single feature of x by a one-hot vector 1i and then compares it to a threshold bi considering a learnable sign si . Each leave returns a one-hot vector of length C as a prediction , where C is the number of classes . Soft decision trees ( SDT ) extend DTs by a ) producing a probabilistic decision utilizing all features of x at each internal node and b ) returning a soft categorical distribution at each leaf as a prediction . The decision functions of these two models are linear with respect to the raw feature x and thus the decision processes are naturally interpretable . Complex models that utilize deep neural networks for decision functions can also be generalized with the decision transformer . Neural decision forests ( NDF ) use a randomized multilayer perceptron ( MLP ) as a decision function . Deep neural decision forests ( DNDF ) use a shared convolutional neural network ( CNN ) for all decisions with changing only the last fully-connected layer . Neural regression forests ( NRF ) improve DNDFs by adopting a hierarchical CNN having different numbers of convolution operations based on the depth of node i in addition to the last fully-connected layer . Other models ( Shen et al. , 2017 ; 2018 ) that have an identical structure to DNDFs and NRFs .

Summary: The authors propose Decision Transformer Networks (DTNs): a model that generalizes decision trees to deep network style decision graphs. Structurally, DTNs are similar to deep neural networks with layers of nodes operating as stochastic decision functions and which output probabilities of an input vector belonging to each sub-network. The probabilistic predictions of the decision nodes in a layer are aggregated in a "transition matrix" and passed to the next layer until the leaf layer is reached where the final classification decision is made. The goal of DTNs is to improve the representation power of tree models by transforming (exponentially large) trees to compact graphs without losing its interpretability power. The authors propose special scoring functions to explain the decisions made by  each layer as well as the model. On a diverse set of datasets ranging from small (<1K) to large (>10K), DTNs (dense and sparse) with softmax and spherical-softmax decision nodes, attained better accuracies than MLPs, conventional decision trees and their soft versions.