Sowing the Seeds of Decision Tree Regression | by Ashwin Raj | Towards Data Science

In this article we will discuss about decision trees, one of the supervised learning algorithm, commonly referred to as CART that can be used for both regression and classification problems.
As the name suggests, the primary role of this algorithm is to make a decision using a tree structure. To find solutions a decision tree makes a sequential, hierarchical decision about the outcome variable based on the predictor data. A decision tree is generally used while working with non-linear data.
Because of their simplicity and the fact that they are easy to understand and implement, they are widely used in a large number of industries.
Now, before we move further it’s important that we understand some important terminologies associated with the algorithm. Decision Trees are made up of a number of nodes, each of which represents a particular feature. The first node of a decision tree is generally referred to as the Root Node.
The depth of the tree is the total number of levels present in the tree excluding the root node. A branch denotes a decision and can be visualized as a link between different nodes. A leaf tells you what class each sample belongs to.
Decision Trees progressively divide data sets into small data groups until they reach sets that are small enough to be described by some label. At the same time an associated decision tree is incrementally developed.
Decision trees apply a top-down approach to data. The splitting of a binary tree can either be binary or multiway. The algorithm partitions the data into a set of rectangles and fits the model over each of these rectangles. More the number of rectangles (splits) greater is the complexity.
A downside of using super complex decision tree is that they are likely of fall into overfitting scenario as the model learns the training data so good that it has problems to generalize to new unseen data.
It then examines all the features of the dataset to find the best possible result by splitting the data into smaller and smaller subgroups until the tree concludes.
The information gain is based on the decrease in entropy after a dataset is split on an attribute. Entropy controls how a decision tree decides to split the data.
The objective of constructing a decision tree is to find the attributes that returns the highest information gain.
Now we will be practically applying what we have learned by building a decision tree model.
You can access the complete code and other resources used for building this decision tree model on my GitHub handle.
The first step in building our model is to import the necessary libraries. Pandas, Numpy and Matplotlib are the most commonly used libraries for data manipulation, scientific computing and plotting data on graphs respectively.
#Import the Libraries and read the data into a Pandas DataFrameimport pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport seaborn as snstest=pd.read_csv("california_housing_test.csv")train=pd.read_csv("california_housing_train.csv")
Seaborn is also being extensively used these days for making statistical graphics in python. After importing the necessary libraries, our next step is to load the dataset into our Jupiter notebook. Here I have used Google collaboratory notebook for demonstration purpose.
After successfully loading the data into our notebook, our next step is to visualize the data. Visualizing the data is important in order to find any correlation between the various features.
#Visualise the dataplt.figure()sns.heatmap(data.corr(), cmap='coolwarm')plt.show()sns.lmplot(x='median_income', y='median_house_value', data=train)sns.lmplot(x='housing_median_age', y='median_house_value', data=train)
After this, appropriate features are selected to build the model. This is commonly referred to as Feature Engineering. Feature engineering is the process of using domain knowledge to extract features from raw data via various data mining techniques.
#Select appropriate featuresdata = data[[‘total_rooms’, ‘total_bedrooms’, ‘housing_median_age’, ‘median_income’, ‘population’, ‘households’]]data.info()data['total_rooms'] = data['total_rooms'].fillna(data['total_rooms'].mean())data['total_bedrooms'] = data['total_bedrooms'].fillna(data['total_bedrooms'].mean()
Feature selection is essentially important when the number of features in our dataset is large.
Once the features are selected, the dataset is split into training data and testing data. This is done by importing the train_test_split function from sklearn library.
#Split the dataset into training and testing dataimport train_test_splitX_train, X_test, y_train, y_test = train_test_split(train, y, test_size = 0.2, random_state = 0)y_train = y_train.reshape(-1,1)y_test = y_test.reshape(-1,1
Note that we have called the library in between our code. Python allows us to import libraries anywhere in between the code.
#Fit the model over training datafrom sklearn.tree import DecisionTreeRegressordt = DecisionTreeRegressor()dt.fit(X_train, y_train)
After this we import the decision tree model from Scikit Learn Library and initialize our regression model. We then fit the data into our model. With that we have succesfully built our decision tree model.
A major problem associated with decision trees is that it suffers from high variance i.e. a small change in data can lead to an entirely different set of splits. Similarly they are also found to create biased trees in scenarios where one class dominates other.
A decision tree is bound within boundaries and generally look for local optimal value instead of the global optimal one. These drawbacks however can be easily corrected using Ensemble methods like bagging and boosting which we will discuss in the upcoming posts.
With this we have reached the end of this article. I hope you would have found this article really informative. If you have any question or if you believe I have made any mistake, please contact me! You can get in touch with me via: Email or LinkedIn.