Neural Regression Tree

1 INTRODUCTION . One of the most challenging and intensively studied problems in machine learning is regression— predicting the continuous value of a dependent variable ( label ) y from the independent variable ( covariates ) x . The relationship between y ∈ Y and x ∈ X , a regression function f : X → Y , is generally unknown and may not be deterministic . Varied approaches can be used to estimate the map f , such as linear regression Bishop ( 2006 ) , nonparametric regression Wasserman ( 2006 ) , neural nets Friedman et al . ( 2001 ) , etc . Among these approaches , one school of methods is regressionvia-classification ( RvC ) Weiss & Indurkhya ( 1995 ) , in which the continuous dependent variable y is discretized into ordered bins and an estimation ŷ is made by classifying the independent variable x into one of the bins . Such class of methods addresses one issue with standard regression methods : different regions of Y may depend on different subsets of X , and a common statistical estimator of f on the entire Y could have high bias on some regions of Y . Therefore , a separate regression function is preferable for each region of Y . In this study , we focus on developing the optimal discretization strategy for RvC . For the sake of a concise explanation , we first define the problem clearly . To start with , we define Partition A partition Π on a set Y is Π ( Y ) = { C1 , . . . , CN } satisfying ⋃N i=1 Ci = Y and Cis are disjoint . When acting on a y ∈ Y , Π ( y ) : = Cj subjected to y ∈ Cj . In our case , Y ⊂ R , and a partition on Y effectively converts the continuous-valued variable y into a categorical one C. This process is often termed as discretization Fayyad & Irani ( 1993 ) ; Dougherty et al . ( 1995 ) . Based on this discretization , a RvC system includes a classifier h : x 7→ { C1 , . . . , CN } and a regressor r : Ci 7→ R. Using this system , a prediction for y is ŷ = r ( h ( x ) ) . The key aspect of an RvC system is its method of partition Π . It uses a set of thresholds T : = { t0 , . . . , tN } to determine Π ( y ) = Cn if tn−1 ≤ y ≤ tn . Problems arise as how to set these thresholds . One approach is to set T by prior knowledge , such as equally probable intervals , equal width intervals , k-means clustering , etc . Dougherty et al . ( 1995 ) ; Torgo & Gama ( 1996 ; 1997 ) . However , this approach could lead to potential difficulties in defining the optimal partition Fayyad & Irani ( 1993 ) , hence the non-ideal performance . For instance , ad-hoc partitions can result in unbalanced partitions that are too easy or too difficult to regress ; partitions that are not fine enough can lead to high regression bias . Therefore , it is necessary to develop strategies to find the optimal discretization of the dependent variable , i.e. , to find the optimal thresholds . We propose a top-down tree based approach , which grows a partition tree with each threshold on the node determined by a neural binary classifier , and the value of dependent variable y is predicted at the leaves . We call the resulting model a neural regression tree ( NRT ) . Specifically , we adopt a top-down binary splitting approach to recursively grow the partition tree . To determine the splitting threshold tn for a node n , we use a binary classifier hn : X→ { −1 , 1 } to find the optimal t∗n by minimizing the classification error t ∗ n = arg mintn Ehn ( Dn , tn ) , where Ehn is the classification error of hn , and Dn is the data on node n. The number of thresholds N , which determines the number of partitions and hence the depth of the tree , stops increasing when the classification performance saturates . Next , in order to determine the value y for any feature x we only need to find out which leave partition x belongs to following the partition tree , and estimate ŷ using a regression function on that bin . The benefits of doing so are three-fold : ( 1 ) the discretization thresholds T are explicitly optimized to obtain optimal partition ; ( 2 ) By adapting a top-down binary splitting approach , we reduce the computational complexity , since directly optimizing T is a hard problem as it scales exponentially with N ; ( 3 ) the features are adapted to each node by the neural classifier , and hence the model promotes hierarchical representation learning , increasing model interpretability . To achieve the above , we propose two algorithms , a scanning search algorithm , and a gradient based method , against a loss function for which we call the triviality loss to solve for the discretization thresholds efficiently . To demonstrate the utility of the proposed approach we conduct experiments on a pair of challenging regression tasks : estimating the age and height of speakers from their voices . Experiments show that our model achieves significant performance gain over state-of-the-art baseline models . 2 RELATED WORK . Regression Trees Tree-structured models have been around for a long time . Among them , the most closely related are the regression trees . Following our terminology , the regression tree can be described as ŷ = r ( Π ( x ) ) , where the partition is performed on independent variable instead of dependent variable . The first regression tree algorithm was presented by Morgan & Sonquist , where they propose a greedy approach to fit a piecewise constant function by recursively splitting the data into two subsets based on partition on X . The optimal split is a result of minimizing the impurity which defines the homogeneity of the split . This algorithm set the basis for a whole line of research on classification and regression trees . Refined algorithms include CART Breiman et al . ( 1984 ) , ID3 Quinlan ( 1986 ) , m5 Quinlan et al . ( 1992 ) , and C4.5 Quinlan ( 2014 ) . Recent work combines the tree-structure and neural nets to gain the power of both structure learning and representation learning . Such work include the convolutional decision trees Laptev & Buhmann ( 2014 ) , neural decision trees Xiao ( 2017 ) ; Balestriero ( 2017 ) , adaptive neural trees Tanno et al . ( 2018 ) , deep neural decision forests Kontschieder et al . ( 2015 ) , and deep regression forests Shen et al . ( 2017 ) . We emphasize that there is a fundamental difference between our approach and the traditional regression tree based approaches : Instead of making the split based on the feature space , our splitting criteria is based on the dependent variables , enabling the features to adapt to the partitions of dependent variables . Regression via Classification ( RvC ) The idea for RvC was presented by Weiss & Indurkhya . Their algorithm was based on k-means clustering to categorize numerical variables . Other conventional approaches Torgo & Gama ( 1996 ; 1997 ) for discretization of continuous values are based on equally probable ( EP ) or equal width ( EW ) intervals , where EP creates a set of intervals with same number of elements , while EW divides into intervals of same range . These approaches are ad-hoc . Instead , we propose a discretization strategy to learn the optimal thresholds by improving the neural classifiers . 3 NEURAL REGRESSION TREE . In this section , we formulate the neural regression tree model for optimal discretization of RvC , and provide algorithms to optimize the model . 3.1 MODEL FORMULATION . Formally , following the description of RvC in Section 1 , a classification rule classifies the data into disjoint bins hθ : x 7→ { C0 , · · · , CN } , where Ci = Π ( y ) corresponds to ti−1 ≤ y ≤ ti , and θ parameterizes the classifier . Then , a regression rule predicts the value of the dependent variable ŷ ( x ) = r ( hθ ( x ) ) , ( 1 ) where r : Ci 7→ [ ti−1 , ti ] is any regression function that operates locally on all instances that are assigned to the bin Ci . Alternatively , the classification rule may compute the probability of a data point x being classified into bin Ci , P ( Ci | x ) = hθ ( x ) , and the regression rule is given by ŷ ( x ) = ECi [ r ( hθ ( x ) ) ] = N∑ i=0 P ( Ci | x ) r ( Ci ) . ( 2 ) Defining an error E ( y , ŷ ( x ) ) between the true y and the estimated ŷ ( x ) , our objective is to learn the thresholds { t0 , . . . , tN } and the parameters { θ1 , . . . , θN } such that the expected error is minimized { t∗n } , { θ∗n } ←− arg min t , θ Ex [ E ( y , ŷ ( x ) ) ] . ( 3 ) Note that the number of thresholds N too is a variable that may either be manually set or explicitly optimized . In practice , instead of minimizing the expected error , we will minimize the empirical average error avg ( E ( yi , ŷ ( xi ) ) ) computed over a training set . However , joint optimization of { tn } and { θn } is a hard problem as it scales exponentially with n. To solve this problem we recast the RvC problem in terms of a binary classification tree , where each of the nodes in the tree is greedily optimized . The structure of the proposed binary tree is shown in Figure 1 . We now describe the tree-growing algorithm . For notational convenience the nodes have been numbered such that for any two nodes n1 and n2 , if n1 < n2 , n1 occurs either to the left of n2 or above it in the tree . Each node n in the tree has an associated threshold tn , which is used to partition the data into its two children n′ and n′′ ( we will assume w.l.o.g . that n′ < n′′ ) . A datum ( x , y ) is assigned to the “ left ” child n′ if y < tn , and to the “ right ” child n′′ otherwise . The actual partitions of the dependent variable are the leaves of the tree . To partition the data , each node carries a classifier hθn : x 7→ { n′ , n′′ } , which assigns any instance with features x to one of n′ or n′′ . In our instantiation of this model , the classifier hθn is a neural classifier that not only classifies the features but also adapts and refines the features to each node . Given an entire tree along with all of its parameters and an input x , we can compute the a posteriori probability of the partitions ( i.e . the leaves ) as follows . For any leaf l , let l0 , · · · , lp represent the chain of nodes from root to leaf , where l0 is the root and lp = l is the leaf itself . The a posteriori probability of the leaf is given by P ( l | x ) = ∏p r=1 P ( lr | lr−1 , x ) , where each P ( lr | lr−1 , x ) is given by the neural classifier on node lr−1 . Substitution into ( 2 ) yields the final regressed value of the dependent variable ŷ ( x ) = ∑ l∈leaves P ( l | x ) r ( l ) , ( 4 ) where r ( l ) , in our setting , is simply the mean value of the leaf bin . Other options include the center of gravity of the leaf bin , using a regression function , etc .

Summary: The paper presents a novel supervised-learning method for regression using decision trees and neural nets. The core idea is based on a 90s technique called "regression via classification" by first apply discretization of target response y by some clustering, and apply any "classification" to those discretized values as class labels. Because real-valued y is one-dimensional and ordered, discretization means setting up any thresholds to give N-partitions of training {y_i}s. The proposed method tries to jointly learn these thresholds as well as node splitters of decision trees using neural nets. Because each node splitters are given by neural nets here, probability outputs for binary classification are also available. Regarding these probabilities as probabilistic splitting at each node, response y weighted by the path probabilities to leaves is the final prediction. The learning is in a greedy manner as in standard tree learning because exact joint learning is computationally hard.  Experiments on speaker profiling illustrate the performance improvements against standard nonlinear regression such as SVR and regression trees.