Oblique Decision Trees from Derivatives of ReLU Networks

1 INTRODUCTION . Decision trees ( Breiman et al. , 1984 ) employ a series of simple decision nodes , arranged in a tree , to transparently capture how the predicted outcome is reached . Functionally , such tree-based models , including random forest ( Breiman , 2001 ) , realize piece-wise constant functions . Beyond their status as de facto interpretable models , they have also persisted as the state of the art models in some tabular ( Sandulescu & Chiru , 2016 ) and chemical datasets ( Wu et al. , 2018 ) . Deep neural models , in contrast , are highly flexible and continuous , demonstrably effective in practice , though lack transparency . We merge these two contrasting views by introducing a new family of neural models that implicitly learn and represent oblique decision trees . Prior work has attempted to generalize classic decision trees by extending coordinate-wise cuts to be weighted , linear classifications . The resulting family of models is known as oblique decision trees ( Murthy et al. , 1993 ) . However , the generalization accompanies a challenging combinatorial , non-differentiable optimization problem over the linear parameters at each decision point . Simple sorting procedures used for successively finding branch-wise optimal coordinate cuts are no longer available , making these models considerably harder to train . While finding the optimal oblique decision tree can be cast as a mixed integer linear program ( Bertsimas & Dunn , 2017 ) , scaling remains a challenge . In this work , we provide an effective , implicit representation of piece-wise constant mappings , termed locally constant networks . Our approach exploits piece-wise linear models such as ReLU networks as basic building blocks . Linearity of the mapping in each region in such models means that the gradient with respect to the input coordinates is locally constant . We therefore implicitly represent piece-wise constant networks through gradients evaluated from ReLU networks . We prove the equivalence between the class of oblique decision trees and these proposed locally constant neural models . However , the sizes required for equivalent representations can be substantially different . For example , a locally constant network with M neurons can implicitly realize an oblique decision tree whose explicit form requires 2M−1 oblique decision nodes . The exponential complexity reduc- 1Our implementation and data are available at https : //github.com/guanghelee/iclr20-lcn . tion in the corresponding neural representation illustrates the degree to which parameters are shared across the locally constant regions . Our locally constant networks can be learned via gradient descent , and they can be explicitly converted back to oblique decision trees for interpretability . For learning via gradient descent , however , it is necessary to employ some smooth annealing of piece-wise linear activation functions so as to keep the gradients themselves continuous . Moreover , we need to evaluate the gradients of all the neurons with respect to the inputs . To address this bottleneck , we devise a dynamic programming algorithm which computes all the necessary gradient information in a single forward pass . A number of extensions are possible . For instance , we can construct approximately locally constant networks by switching activation functions , or apply helpful techniques used with normal deep learning models ( e.g. , DropConnect ( Wan et al. , 2013 ) ) while implicitly training tree models . We empirically test our model in the context of molecular property classification and regression tasks ( Wu et al. , 2018 ) , where tree-based models remain state-of-the-art . We compare our approach against recent methods for training oblique decision trees and classic ensemble methods such as gradient boosting ( Friedman , 2001 ) and random forest . Empirically , a locally constant network always outperforms alternative methods for training oblique decision trees by a large margin , and the ensemble of locally constant networks is competitive with classic ensemble methods . 2 RELATED WORK . Locally constant networks are built on a mixed integer linear representation of piece-wise linear networks , defined as any feed-forward network with a piece-wise linear activation function such as ReLU ( Nair & Hinton , 2010 ) . One can specify a set of integers encoding the active linear piece of each neuron , which is called an activation pattern ( Raghu et al. , 2017 ) . The feasible set of an activation pattern forms a convex polyhedron in the input space ( Lee et al. , 2019 ) , where the network degenerates to a linear model . The framework motivates us to leverage the locally invariant derivatives of the networks to construct a locally constant network . The activation pattern is also exploited in literature for other purposes such as deriving robustness certificates ( Weng et al. , 2018 ) . We refer the readers to the recent work ( Lee et al. , 2019 ) and the references therein . Locally constant networks use the gradients of deep networks with respect to inputs as the representations to build discriminative models . Such gradients have been used in literature for different purposes . They have been widely used for local sensitivity analysis of trained networks ( Simonyan et al. , 2013 ; Smilkov et al. , 2017 ) . When the deep networks model an energy function ( LeCun et al. , 2006 ) , the gradients can be used to draw samples from the distribution specified by the normalized energy function ( Du & Mordatch , 2019 ; Song & Ermon , 2019 ) . The gradients can also be used to train generative models ( Goodfellow et al. , 2014 ) or perform knowledge distillation ( Srinivas & Fleuret , 2018 ) . The class of locally constant networks is equivalent to the class of oblique decision trees . There are some classic methods that also construct neural networks that reproduce decision trees ( Sethi , 1990 ; Brent , 1991 ; Cios & Liu , 1992 ) , by utilizing step functions and logic gates ( e.g. , AND/NEGATION ) as the activation function . The methods were developed when back-propagation was not yet practically useful , and the motivation is to exploit effective learning procedures of decision trees to train neural networks . Instead , our goal is to leverage the successful deep models to train oblique decision trees . Recently , Yang et al . ( 2018 ) proposed a network architecture with arg max activations to represent classic decision trees with coordinate cuts , but their parameterization scales exponentially with input dimension . In stark contrast , our parameterization only scales linearly with input dimension ( see our complexity analyses in §3.7 ) . Learning oblique decision trees is challenging , even for a greedy algorithm ; for a single oblique split , there can be ∑D k=0 ( N k ) different ways to separate N data points in D-dimensional space ( Vapnik & Chervonenkis , 1971 ) ( cf . ND possibilities for coordinate-cuts ) . Existing learning algorithms for oblique decision trees include greedy induction , global optimization , and iterative refinements on an initial tree . We review some representative works , and refer the readers to the references therein . Optimizing each oblique split in greedy induction can be realized by coordinate descent ( Murthy et al. , 1994 ) or a coordinate-cut search in some linear projection space ( Menze et al. , 2011 ; Wickramarachchi et al. , 2016 ) . However , the greedy constructions tend to get stuck in poor local optimum . There are some works which attempt to find the global optimum given a fixed tree structure by formulating a linear program ( Bennett , 1994 ) or a mixed integer linear program ( Bertsimas & Dunn , 2017 ) , but the methods are not scalable to ordinary tree sizes ( e.g. , depth more than 4 ) . The iterative refinements are more scalable than global optimization , where CART ( Breiman et al. , 1984 ) is the typical initialization . Carreira-Perpinán & Tavallali ( 2018 ) develop an alternating optimization method via iteratively training a linear classifier on each decision node , which yield the state-of-theart empirical performance , but the approach is only applicable to classification problems . Norouzi et al . ( 2015 ) proposed to do gradient descent on a sub-differentiable upperbound of tree prediction errors , but the gradients with respect to oblique decision nodes are unavailable whenever the upperbound is tight . In contrast , our method conducts gradient descent on a differentiable relaxation , which is gradually annealed to a locally constant network . 3 METHODOLOGY . In this section , we introduce the notation and basics in §3.1 , construct the locally constant networks in §3.2-3.3 , analyze the networks in §3.4-3.5 , and develop practical formulations and algorithms in §3.6-3.7 . Note that we will propose two ( equivalent ) architectures of locally constant networks in §3.3 and §3.6 , which are useful for theoretical analyses and practical purposes , respectively . 3.1 NOTATION AND BASICS . The proposed approach is built on feed-forward networks that yield piece-wise linear mappings . Here we first introduce a canonical example of such networks , and elaborate its piece-wise linearity . We consider the densely connected architecture ( Huang et al. , 2017 ) , where each hidden layer takes as input all the previous layers ; it subsumes other existing feed-forward architectures such as residual networks ( He et al. , 2016 ) . For such a network fθ : RD → RL with the set of parameters θ , we denote the number of hidden layers asM and the number of neurons in the ith layer asNi ; we denote the neurons in the ith layer , before and after activation , as zi ∈ RNi and ai ∈ RNi , respectively , where we sometimes interchangeably denote the input instance x as a0 ∈ RN0 with N0 , D. To simplify exposition , we denote the concatenation of ( a0 , a1 , . . . , ai ) as ãi ∈ RÑi with Ñi , ∑i j=0Ni , ∀i ∈ { 0 , 1 , . . . , M } . The neurons are defined via the weight matrix W i ∈ RNi×Ñi−1 and the bias vector bi ∈ RNi in each layer i ∈ [ M ] , { 1 , 2 , . . . , M } . Concretely , a0 , x , zi , W iãi−1 + bi , ai , σ ( zi ) , ∀i ∈ [ M ] , ( 1 ) where σ ( · ) is a point-wise activation function . Note that both a and z are functions of the specific instance denoted by x , where we drop the functional dependency to simplify notation . We use the set I to denote the set of all the neuron indices in this network { ( i , j ) |j ∈ [ Ni ] , i ∈ [ M ] } . In this work , we will use ReLU ( Nair & Hinton , 2010 ) as a canonical example for the activation function aij = σ ( z i ) j , max ( 0 , z i j ) , ∀ ( i , j ) ∈ I , ( 2 ) but the results naturally generalize to other piece-wise linear activation functions such as leaky ReLU ( Maas et al. , 2013 ) . The output of the entire network fθ ( x ) is the affine transformation from all the hidden layers ãM with the weight matrix WM+1 ∈ RL×ÑM and bias vector bM+1 ∈ RL . 3.2 LOCAL LINEARITY . It is widely known that the class of networks fθ ( · ) yields a piece-wise linear function . The results are typically proved via associating the end-to-end behavior of the network with its activation pattern – which linear piece in each neuron is activated ; once an activation pattern is fixed across the entire network , the network degenerates to a linear model and the feasible set with respect to an activation pattern is a natural characterization of a locally linear region of the network . Formally , we define the activation pattern as the collection of activation indicator functions for each neuron oij : RD → { 0 , 1 } , ∀ ( i , j ) ∈ I ( or , equivalently , the derivatives of ReLU units ; see below ) 2 : oij = ∂aij ∂zij , I [ zij ≥ 0 ] , ∀ ( i , j ) ∈ I , ( 3 ) 2Note that each oij is again a function of x , where we omit the dependency for brevity . 𝐷𝑒𝑓𝑖𝑛𝑒 𝑧 # = 𝑥 # − 𝑥 & + 1 ; 𝑧 & = −4𝑥 # + 𝑥 & + 4𝑎 # + 4 ; 𝑧 ' = −𝑥 # + 𝑥 & + 1.5𝑎 # − 0.3𝑎 & − 3. where I [ · ] is the indicator function . Note that , for mathematical correctness , we define ∂aij/∂zij = 1 at zij = 0 ; this choice is arbitrary , and one can change it to ∂a i j/∂z i j = 0 at z i j = 0 without affecting most of the derivations . Given a fixed activation pattern ōij ∈ { 0 , 1 } , ∀ ( i , j ) , we can specify a feasible set in RD that corresponds to this activation pattern { x ∈ RD|oij = ōij , ∀ ( i , j ) ∈ I } ( note that each oij is a function of x ) . Due to the fixed activation pattern , the non-linear ReLU can be re-written as a linear function for all the inputs in the feasible set . For example , for an ōij = 0 , we can re-write aij = 0×zij . As a result , the network has a consistent end-to-end linear behavior across the entire feasible set . One can prove that all the feasible sets partition the space RD into disjoint convex polyhedra3 , which realize a natural representation of the locally linear regions . Since we will only use the result to motivate the construction of locally constant networks , we refer the readers to Lee et al . ( 2019 ) for a detailed justification of the piece-wise linearity of such networks .

Summary: This paper proposes locally constant network (LCN), which is implemented via the gradient of piece-wise linear networks such as ReLU networks. The authors built the equivalence between LCN and decision trees, and also demonstrated that LCN with M neurons has the same representation capability as decision trees with 2^M leaf nodes. The experiments conducted in the paper disclose that training LCN outperforms other methods using decision trees.