A Neural Tangent Kernel Perspective of Infinite Tree Ensembles

1 INTRODUCTION . Tree ensembles and neural networks are powerful machine learning models used in various realworld applications . A soft tree ensemble is one of the variants of tree ensemble models that inherits characteristics of neural networks . Instead of using a greedy method ( Quinlan , 1986 ; Breiman et al. , 1984 ) for searching splitting rules , the soft tree makes the splitting rules soft and updates the entire model ’ s parameters simultaneously using the gradient method . Soft tree ensemble models are known to have high empirical performance ( Kontschieder et al. , 2015 ; Popov et al. , 2020 ; Hazimeh et al. , 2020 ) , especially for tabular datasets . Besides accuracy , there are many additional reasons why one should formulate trees in a soft manner . For example , unlike hard decision trees , the model can be updated sequentially ( Ke et al. , 2019 ) and trained in combination with pre-training ( Arik & Pfister , 2019 ) , resulting in favorable characteristics in terms of real-world continuous service deployment . Their model interpretability induced by the hierarchical splitting structure has also attracted much attention ( Frosst & Hinton , 2017 ; Wan et al. , 2021 ; Tanno et al. , 2019 ) . In addition , the idea of the soft tree is implicitly used in many different places ; for example , the process of allocating data to the appropriate leaves can be interpreted as a special case of Mixture-of-Experts ( Jordan & Jacobs , 1993 ; Shazeer et al. , 2017 ; Lepikhin et al. , 2021 ) , a technique for balancing computational complexity and prediction performance . Although various techniques have been proposed to train trees , the theoretical validity of such techniques is not well understood at sufficient depth . Examples of the practical technique include constraints on individual trees using parameter sharing ( Popov et al. , 2020 ) , adjusting the hardness of the splitting operation ( Frosst & Hinton , 2017 ; Hazimeh et al. , 2020 ) , and the use of overparameterization ( Belkin et al. , 2019 ; Karthikeyan et al. , 2021 ) . To better understand the training of tree ensemble models , we focus on the Neural Tangent Kernel ( NTK ) ( Jacot et al. , 2018 ) , a powerful tool that has been successfully applied to various neural network models with infinite hidden layer nodes . Every model architecture is known to produce a distinct NTK . Not only for the multi-layer perceptron ( MLP ) , many studies have been performed across various models , such as for Convolutional Neural Networks ( CNTK ) ( Arora et al. , 2019 ; Li et al. , 2019 ) , Graph Neural Networks ( GNTK ) ( Du et al. , 2019b ) , and Recurrent Neural Networks ( RNTK ) ( Alemohammad et al. , 2021 ) . The NTK theory is often used in the context of overparameterization of neural networks . In response to recent trends , overparameterization is also a subject of interest for tree ensembles ( Belkin et al. , 2019 ; Karthikeyan et al. , 2021 ) . Although a number of findings have been obtained using the NTK , they are mainly for typical neural networks , and it is still not obvious how to apply the NTK theory to the tree models . In this paper , by considering the limit of infinitely many trees , we introduce and study the neural tangent kernel for tree ensembles , called the Tree Neural Tangent Kernel ( TNTK ) , which provides new insights into the behavior of the ensemble of soft trees . The goal of this research is to derive the kernel that characterizes the training behavior of soft tree ensembles , and to obtain theoretical support for the empirical techniques . Our contributions are summarized as follows : • First extension of the NTK concept to the tree ensemble models . We derive the analytical form for the TNTK at initialization induced by infinitely many complete binary trees with arbitrary depth . ( Section 4.1.1 ) . We also prove that the TNTK remains constant during the training of infinite soft trees . This property allows us to analyze the behavior by kernel regression and discuss global convergence of training using the positive definiteness of the TNTK ( Section 4.1.2 , 4.1.3 ) . • Equivalence of the oblivious tree ensemble models . We show the TNTK induced by the oblivious tree structure used in practical open-source libraries such as CatBoost ( Prokhorenkova et al. , 2018 ) and NODE ( Popov et al. , 2020 ) converges to the same TNTK induced by a non-oblivious one in the limit of infinite trees . This observation implicitly supports the good empirical performance of oblivious trees with parameter sharing between tree nodes ( Section 4.2.1 ) . • Nonlinearity by adjusting the tree splitting operation . Practically , various functions have been proposed to represent the tree splitting operation . The most basic function is sigmoid . We show that the TNTK is almost a linear kernel in the basic case , and when we adjust the splitting function hard , the TNTK becomes nonlinear ( Section 4.2.2 ) . • Degeneracy of the TNTK with deep trees . The TNTK associated with deep trees exhibits degeneracy : the TNTK values are almost identical for deep trees even if the inner products of inputs are different . As a result , poor performance in numerical experiments is observed with the TNTK induced by infinitely many deep trees . This result supports the fact that the depth of trees is usually not so large in practical situations ( Section 4.2.3 ) . • Comparison to the NTK induced by the MLP . We investigate the generalization performance of infinite tree ensembles by kernel regression with the TNTK on 90 real-world datasets . Although the MLP with infinite width has better prediction accuracy on average , the infinite tree ensemble performs better than the infinite width MLP in more than 30 % of the datasets . We also showed that the TNTK is superior to the MLP-induced NTK in computational speed ( Section 5 ) . 2 BACKGROUND AND RELATED WORK . Our main focus in this paper is the soft tree and the neural tangent kernel . We briefly introduce and review them . 2.1 SOFT TREE . Based on Kontschieder et al . ( 2015 ) , we formulate a regression by soft trees . Figure 1 is a schematic image of an ensemble of M soft trees . We define a data matrix x ∈ RN0×N for N training samples { x1 , . . . , xN } with N0 features and define tree-wise parameter matrices for internal nodes wm ∈ RN0×N and leaf nodes πm ∈ R1×L for each tree m ∈ [ M ] = { 1 , . . . , M } as x = ( | . . . | x1 . . . xN | . . . | ) , wm = ( | . . . | wm,1 . . . wm , N | . . . | ) , πm = ( πm,1 , . . . , πm , L ) , where internal nodes ( blue nodes in Figure 1 ) and leaf nodes ( green nodes in Figure 1 ) are indexed from 1 to N and 1 to L , respectively . N and L may change across trees in general , while we assume that they are always fixed for simplicity throughout the paper . We also write horizontal concatenation of ( column ) vectors as x = ( x1 , . . . , xN ) ∈ RN0×N and wm = ( wm,1 , . . . , wm , N ) ∈ RN0×N . Unlike hard decision trees , we consider a model in which every single leaf node ` ∈ [ L ] = { 1 , . . . , L } of a tree m holds the probability that data will reach to it . Therefore , splitting operation at an intermediate node n ∈ [ N ] = { 1 , . . . , N } does not definitively decide splitting to the left or right . To provide an explicit form of the probabilistic tree splitting operation , we introduce the following binary relations that depend on the tree ’ s structure : ` ↙ n ( resp . n ↘ ` ) , which is true if a leaf ` belongs to the left ( resp . right ) subtree of a node n and false otherwise . We can now exploit µm , ` ( xi , wm ) : RN0 ×RN0×N → [ 0 , 1 ] , a function that returns the probability that a sample xi will reach a leaf ` of the tree m , as follows : µm , ` ( xi , wm ) = N∏ n=1 gm , n ( xi , wm , n ) 1 ` ↙n ( 1− gm , n ( xi , wm , n ) ) 1n↘ ` , ( 1 ) where 1Q is an indicator function conditioned on the argument Q , i.e. , 1true = 1 and 1false = 0 , and gm , n : RN0×RN0 → [ 0 , 1 ] is a decision function at each internal node n of a treem . To approximate decision tree splitting , the output of the decision function gm , n should be between 0.0 and 1.0 . If the output of a decision function takes only 0.0 or 1.0 , the splitting operation is equivalent to hard splitting used in typical decision trees . We will define an explicit form of the decision function gm , n in Equation ( 5 ) in the next section . The prediction for each xi from a tree m with nodes parameterized by wm and πm is given by fm ( xi , wm , πm ) = L∑ ` =1 πm , ` µm , ` ( xi , wm ) , ( 2 ) where fm : RN0 × RN0×N × R1×L → R , and πm , ` denotes the response of a leaf ` of the tree m. This formulation means that the prediction output is the average of the leaf values πm , ` weighted by µm , ` ( xi , wm ) , probability of assigning the sample xi to the leaf ` . If µm , ` ( xi , wm ) takes only 1.0 for one leaf and 0.0 for the other leaves , the behavior is equivalent to a typical decision tree prediction . In this model , wm and πm are updated during training with a gradient method . While many empirical successes have been reported , theoretical analysis for soft tree ensemble models has not been sufficiently developed . 2.2 NEURAL TANGENT KERNEL . Given N samples x ∈ RN0×N , the NTK induced by any model architecture at a training time τ is formulated as a matrix Ĥ∗τ ∈ RN×N , in which each ( i , j ) ∈ [ N ] × [ N ] component is defined as [ Ĥ∗τ ] ij : = Θ̂ ∗ τ ( xi , xj ) : = 〈 ∂farbitrary ( xi , θτ ) ∂θτ , ∂farbitrary ( xj , θτ ) ∂θτ 〉 , ( 3 ) where 〈· , ·〉 denotes the inner product and θτ ∈ RP is a concatenated vector of all the P trainable model parameters at τ . An asterisk “ ∗ ” indicates that the model is arbitrary . The model function farbitrary : RN0 × RP → R used in Equation ( 3 ) is expected to be applicable to a variety of model structures . For the soft tree ensembles introduced in Section 2.1 , the NTK is formulated as∑M m=1 ∑N n=1 〈 ∂f ( xi , w , π ) ∂wm , n , ∂f ( xj , w , π ) ∂wm , n 〉 + ∑M m=1 ∑L ` =1 〈 ∂f ( xi , w , π ) ∂πm , ` , ∂f ( xj , w , π ) ∂πm , ` 〉 . In the limit of infinite width with a proper parameter scaling , a variety of properties have been discovered from the NTK induced by the MLP . For example , Jacot et al . ( 2018 ) showed the convergence of Θ̂MLP0 ( xi , xj ) , which can vary with respect to parameters , to the unique limiting kernel Θ ( xi , xj ) at initialization in probability . Moreover , they also showed that the limiting kernel does not change during training in probability : lim width→∞ Θ̂MLPτ ( xi , xj ) = limwidth→∞ Θ̂MLP0 ( xi , xj ) = : Θ MLP ( xi , xj ) . ( 4 ) This property helps in the analytical understanding of the model behavior . For example , with the squared loss and infinitesimal step size with learning rate η , the training dynamics of gradient flow in function space coincides with kernel ridge-less regression with the limiting NTK . Such a property gives us a data-dependent generalization bound ( Bartlett & Mendelson , 2003 ) related to the NTK and the prediction targets . In addition , if the NTK is positive definite , the training can achieve global convergence ( Du et al. , 2019a ; Jacot et al. , 2018 ) . Although a number of findings have been obtained using the NTK , they are mainly for typical neural networks such as MLP and ResNet , and the NTK theory has not been applied to tree models yet . The NTK theory is often used in the context of overparameterization , a subject of interest not only for the neural networks but also for the tree models ( Belkin et al. , 2019 ; Karthikeyan et al. , 2021 ; Tang et al. , 2018 ) . 3 SETUP We train model parametersw and π to minimize the squared loss using the gradient method , where w = ( w1 , . . . , wm ) and π = ( π1 , . . . , πm ) . The tree structure is fixed during training . In order to use a known closed-form solution of the NTK ( Williams , 1996 ; Lee et al. , 2019 ) , we use a scaled error function σ : R→ ( 0 , 1 ) , resulting in the following decision function : gm , n ( xi , wm , n ) = σ ( w > m , nxi ) : = 1 2 erf ( αw > m , nxi ) + 1 2 , ( 5 ) where erf ( p ) = 2√ π ∫ p 0 e−t 2 dt for p ∈ R. This scaled error function approximates commonly used sigmoid function . Since the bias term for the input of σ can be expressed inside of w by adding an element that takes a fixed constant value for all input of the soft trees x , we do not consider the bias for simplicity . The scaling factor α is introduced by Frosst & Hinton ( 2017 ) to avoid too soft splitting . Figure 2 shows that the decision function becomes harder as α increases ( from blue to red ) , and in the limit α→∞ it coincides with the hard splitting used in typical decision trees . When aggregating the output of multiple trees , we divide the sum of the tree outputs by the square root of the number of trees f ( xi , w , π ) = 1√ M M∑ m=1 fm ( xi , wm , πm ) . ( 6 ) This 1/ √ M scaling is known to be essential in the existing NTK literature to use the weak law of the large numbers ( Jacot et al. , 2018 ) . On top of Equation ( 6 ) , we initialize each of model parameters wm , n and πm , ` with zero-mean i.i.d . Gaussians with unit variances . We refer such a parameterization as NTK initialization . In this paper , we consider a model such that allM trees have the same complete binary tree structure , a common setting for soft tree ensembles ( Popov et al. , 2020 ; Kontschieder et al. , 2015 ; Hazimeh et al. , 2020 ) .

Summary: The paper presents an extension of a neural network analytic tool called Neural Tangent Kernels to its use for decision trees and forests.  The focus is a previously proposed notion of soft trees. A few analytical results are presented about the properties of the extended notion, called Tree Neural Tangent Kernel, such as the existence of some limiting deterministic form, how alternative probabilistic tree ensembles would have their kernels converging to this deterministic form, that this limiting kernel is positive definite, and that the kernel remains stable during training, equivalence of the kernels for one-level tree ensembles to those of two-layer perceptrons, etc. Some support for the analytical claims is provided by simulation experiments.