Decision Tree Regression
By: Navjot Singh
Tags: Machine Learning, Python, Decision Tree, Data Analysis, Data Science

Before we begin with the Regression Trees or the Decision Tree Regression, let’s recall and review the simple Decision Tree.

Decision Tree is a supervised machine learning algorithm and it is one of the popular machine learning algorithm. It is a tree like structure constructed on the basis of attributes/features . Decision Trees is the non-parametric supervised learning approach.

So guys you may have heard the term CART which stands for Classification and Regression Trees. In my pervious blog of Decision Tree,I have covered basics of Decision Tree and Classification tree. Regression Trees are bit more complex than classification tree.

Decision Tree Regression is a Non- Linear Regression technique.

Decision Tree Regression model is in the form of a tree structure.It breaks down a data set into smaller and smaller subsets while at the same time an associated decision tree is developed.

Decision Tree Regression is used to predict the target variable whose values are continous in nature. Regression Tree can easily handle the complicated data.

Impurity Measure for regression tree are :

Least Square : Least squares regression is a way to find the line of best fit for a set of data. It does this by creating a model that minimizes the sum of the squared vertical distances (residuals).

: Least squares regression is a way to find the line of best fit for a set of data. It does this by creating a model that minimizes the sum of the squared vertical distances (residuals). Least absolute deviations:Least absolute deviations attempts to find a function which closely approximates a set of data. Instead of minimizing the sum of squared errors, it minimizes the sum of absolute values of errors.

So let us consider an example and implement the decision tree regression, here we have a dataset that has been given to us and we’ve got a scatterplot which represent it.

In the dataset we got two independent variable X1 and X2 and we have to predict the third variable a dependent variable.In the plot we cannot see it because this a simple two dimensional chart only fit the two variable,Y is the third dimension which look like this.

So once we run the regression tree or decision tree regression the scatterplot will be split up into many segments and the splits are create by an algorithm. How and where these splits are conducting is determined by the algorithm.

The right split is the split which increases the amount of information and algorithm knows when to stop when there is certain minimum amount of information that needs to be added and once they cannot add more information to the setup by splitting or we have less then 5% of the total point in the leaf then that is the leaf node and no further split take place.

Then these optimal splits will help us to create the decision tree. As according to the first split is at 20(X1),second at 170(X2) on the right side of first split,third at 200(X2) on the left side of the first split and fourth at 40(X1) below the second split. So as we were splitting the data and adding information to the system and the information falls into the terminal leaf. The Information helps us to predict the value for the new element and it works just by taking the averages of each of the segments.

In Rregression Trees, each leaf node represents a numeric value

Splitting in Decision Tree

In order to determine how to split the data into groups or segments, we try to find the different thresholds that gave us the smallest sum of squared residuals.The splits are chosen to minimize the residual sum of squares between the observation and the mean in each node. Mathematical formula of residual as follow

The threshold with the smallest sum of squared residuals becomes a candidate for the root of the tree. Mathematical formula of RSS (residual sum of squares) as follow

In order to find out the best split, we must minimize the RSS

If we have more than one predictor i.e. independent variables,we find the optimal threshold for each one and pick the candidate with the smallest sum of squared residuals to be the root.

When we cannot add more amount of information to the system and we have less than 5% of the total points in the leaf then that node becomes a leaf node, otherwise we repeat the process to split the remaning observation untill we can no longer split the observation into smaller groups or segment then we are done.

Regression Tree can easily accommodated the additional predictors.

That’s all for Regression in Decision Tree Machine learning Algorithm. Stay tuned for further blogs.

Thankyou

Implementation of Decision tree Regression algorithm on Position-Salaries dataset ~

Dataset : Position_Salaries dataset

Link:https://github.com/InternityFoundation/MachineLearning_Navu4/blob/master/Day%208%20:%20Decision%20Tree/Decision%20Tree%20Regression%20.ipynb