Decision Tree. In our day-to-day life, we interact... | by Aditya Kumar | Towards Data Science

In our day-to-day life, we interact with various machine learning applications and use it without knowing it. The best example is buying something from any online shopping portal where we get several recommendations based on what we are buying.
One type of machine learning algorithm is Decision Tree, which is a type of classification algorithm that comes under supervised classification.
The decision tree is something that we might have used knowingly or unknowingly. Consider the case of buying a car. We will choose a car after considering various factors like budget, safety, color, and price. We first checked the price which is less than X followed by Color followed by safety and then reached a conclusion.
Looking at the above diagram we can define the Decision tree is a graphical representation of a tree-shaped diagram that is used to determine the course of action. Each branch of the tree represents a decision.
The problem decision tree can solve?
It can solve two types of problems.
Classification: Classify based on if-then condition. Ex: If a flower color is red then its rose, if it’s white then lily.Regression: Regression tree is used when there is continuous data.
Classification: Classify based on if-then condition. Ex: If a flower color is red then its rose, if it’s white then lily.
Regression: Regression tree is used when there is continuous data.
Advantages of Decision tree
Simple to understand.Little effort in data preparation.The non-linear parameter does not affect performance.
Simple to understand.
Little effort in data preparation.
The non-linear parameter does not affect performance.
Disadvantages:
Overfitting: In the case of noise in the data.Instability: Models can become unstable due to variations in data.
Overfitting: In the case of noise in the data.
Instability: Models can become unstable due to variations in data.
Important terminology to be used in the Decision Tree.
Entropy: It’s the measure of unpredictability in the dataset. For example, we have a bucket of fruits. Here everything is mixed and hence it’s entropy is very high.Information gain: There’s a decrease in the entropy. For example, if we have a bucket of 5 different fruits. If all are kept in one place then the information gained is minimal. But if we keep all 5 fruits separate we see the entropy as min as it’s not mixed and information gained as maximum.Leaf node: It’s the end of the decision tree that carries the information. In the figure above we can say “Buy” is the leaf node.Decision Node: It’s the mid node in the decision tree where 2 or more new splits arise. In the above diagram, color is a decision node because it further splits into red and blue.Root Node: It’s the topmost node of the figure where all the information is stored or has the highest entropy. In the diagram “Car” is the root node.
Entropy: It’s the measure of unpredictability in the dataset. For example, we have a bucket of fruits. Here everything is mixed and hence it’s entropy is very high.
Information gain: There’s a decrease in the entropy. For example, if we have a bucket of 5 different fruits. If all are kept in one place then the information gained is minimal. But if we keep all 5 fruits separate we see the entropy as min as it’s not mixed and information gained as maximum.
Leaf node: It’s the end of the decision tree that carries the information. In the figure above we can say “Buy” is the leaf node.
Decision Node: It’s the mid node in the decision tree where 2 or more new splits arise. In the above diagram, color is a decision node because it further splits into red and blue.
Root Node: It’s the topmost node of the figure where all the information is stored or has the highest entropy. In the diagram “Car” is the root node.
As discussed, the decision tree can be used in two scenarios. Let’s check the usage in a regression environment.
Decision Tree: Regression
import pandas as pdimport numpy as npimport matplotlib.pyplot as plt%matplotlib inlinedf = pd.read_csv("../input/automobileEDA.csv")df.head()
df.head() will give us the details of the top 5 rows of every column. We can use df.tail() to get the last 5 rows and similarly df.head(10) to get to the top 10 rows.
The data is about cars, and we need to predict the price of the car using the above data
We will be using the Decision Tree to predict the price of the car.
df.dtypes
Out:
symboling              int64normalized-losses      int64make                  objectaspiration            objectnum-of-doors          objectbody-style            objectdrive-wheels          objectengine-location       objectwheel-base           float64length               float64width                float64height               float64curb-weight            int64engine-type           objectnum-of-cylinders      objectengine-size            int64fuel-system           objectbore                 float64stroke               float64compression-ratio    float64horsepower           float64peak-rpm             float64city-mpg               int64highway-mpg            int64price                float64city-L/100km         float64horsepower-binned     objectdiesel                 int64gas                    int64dtype: object
dtypes give the data type of column.
df.describe()
In the above data frame, the column data type is a mix of object, float, and int64. So we will consider only those columns whose values are in numeric and will make all numeric to float.
df.dtypesfor x in df:    if df[x].dtypes == "int64":        df[x] = df[x].astype(float)        print (df[x].dtypes)float64float64float64float64float64float64float64float64
Preparing the Data.
As with the classification task, in this section, we will divide our data into attributes and labels and consequently into training and test sets. We will create 2 data sets, one for the price while the other (df-price). Since our data frame has a lot of data in object format, for this analysis we are removing all the columns with object type and for all NaN values, we will remove that row.
df = df.select_dtypes(exclude=['object'])df=df.fillna(df.mean())X = df.drop('price',axis=1)y = df['price']
Here the X variable contains all the columns from the dataset, except the ‘Price’ column, which is the label. The y variable contains values from the ‘Price’ column, which means that the X variable contains the attribute set and y variable contains the corresponding labels.
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
To train the algorithm, we will use the DecisionTreeRegressor class and the fit method.
from sklearn.tree import DecisionTreeRegressorregressor = DecisionTreeRegressor()regressor.fit(X_train, y_train)
Let’s predict the price.
y_pred = regressor.predict(X_test)
Let’s check the difference between actual and predicted values.
df=pd.DataFrame({'Actual':y_test, 'Predicted':y_pred})df
from sklearn import metricsprint('Mean Absolute Error:', metrics.mean_absolute_error(y_test, y_pred))print('Mean Squared Error:', metrics.mean_squared_error(y_test, y_pred))print('Root Mean Squared Error:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))
import seaborn as snsplt.figure(figsize=(5, 7))ax = sns.distplot(y, hist=False, color="r", label="Actual Value")sns.distplot(y_pred, hist=False, color="b", label="Fitted Values" , ax=ax)plt.title('Actual vs Fitted Values for Price')plt.xlabel('Price (in dollars)')plt.ylabel('Proportion of Cars')plt.show()plt.close()
The above is the graph between the actual and predicted values.
Decision Tree: Classification
We will use the Decision tree classification to predict the results from our model.
Let’s import the library needed.
import numpy as np import pandas as pdfrom sklearn.tree import DecisionTreeClassifier
About the data:
The data is about a list of patients with a list of diseases. Through the model, we will predict which drug should be provided to the patient.
df = pd.read_csv("../input/drug200.csv")df.head()
Calculate the size of the data.
np.size(df)
Out:
1200
Preparing the data
We will take some of the columns that are needed for the analysis.
X = df[['Age', 'Sex', 'BP', 'Cholesterol', 'Na_to_K']].valuesX[0:5]
Out:
array([[23, 'F', 'HIGH', 'HIGH', 25.355],       [47, 'M', 'LOW', 'HIGH', 13.093],       [47, 'M', 'LOW', 'HIGH', 10.113999999999999],       [28, 'F', 'NORMAL', 'HIGH', 7.797999999999999],       [61, 'F', 'LOW', 'HIGH', 18.043]], dtype=object)
In the above array, some values in the dataset are not numeric such as Sex or BP. Unfortunately, Sklean Decision Trees cannot handle these variables. Hence we need to convert the same into numeric. pandas.get_dummies() Convert non-numeric variables into dummy/indicator variables.
from sklearn import preprocessingle_sex = preprocessing.LabelEncoder()le_sex.fit(['F','M'])X[:,1] = le_sex.transform(X[:,1]) le_BP = preprocessing.LabelEncoder()le_BP.fit([ 'LOW', 'NORMAL', 'HIGH'])X[:,2] = le_BP.transform(X[:,2])le_Chol = preprocessing.LabelEncoder()le_Chol.fit([ 'NORMAL', 'HIGH'])X[:,3] = le_Chol.transform(X[:,3]) X[0:5]
Out:
array([[23, 0, 0, 0, 25.355],       [47, 1, 1, 0, 13.093],       [47, 1, 1, 0, 10.113999999999999],       [28, 0, 2, 0, 7.797999999999999],       [61, 0, 1, 0, 18.043]], dtype=object)
Above we have changed the values to 0,1,2 as numeric.
#target valuey = df["Drug"]y
Out:
0      drugY1      drugC2      drugC3      drugX4      drugY       ...  195    drugC196    drugC197    drugX198    drugX199    drugXName: Drug, Length: 200, dtype: object
We will be using the train/test split on our decision tree. Let’s import train_test_split from sklearn.cross_validation.
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=3)
We will create a test and train data
We will first create an instance of the DecisionTreeClassifier called Tree.
Tree = DecisionTreeClassifier(criterion="entropy", max_depth = 4)Tree # it shows the default parameters
Out:
DecisionTreeClassifier(ccp_alpha=0.0, class_weight=None, criterion='entropy',                       max_depth=4, max_features=None, max_leaf_nodes=None,                       min_impurity_decrease=0.0, min_impurity_split=None,                       min_samples_leaf=1, min_samples_split=2,                       min_weight_fraction_leaf=0.0, presort='deprecated',                       random_state=None, splitter='best')
Let’s predict our values.
pred =Tree.predict(X_test)pred
Out
array(['drugY', 'drugX', 'drugX', 'drugX', 'drugX', 'drugC', 'drugY',       'drugA', 'drugB', 'drugA', 'drugY', 'drugA', 'drugY', 'drugY',       'drugX', 'drugY', 'drugX', 'drugX', 'drugB', 'drugX', 'drugX',       'drugY', 'drugY', 'drugY', 'drugX', 'drugB', 'drugY', 'drugY',       'drugA', 'drugX', 'drugB', 'drugC', 'drugC', 'drugX', 'drugX',       'drugC', 'drugY', 'drugX', 'drugX', 'drugX', 'drugA', 'drugY',       'drugC', 'drugY', 'drugA', 'drugY', 'drugY', 'drugY', 'drugY',       'drugY', 'drugB', 'drugX', 'drugY', 'drugX', 'drugY', 'drugY',       'drugA', 'drugX', 'drugY', 'drugX'], dtype=object)
Let’s check the test and predict the value.
print (pred [0:5])print (y_test [0:5])
Out:
['drugY' 'drugX' 'drugX' 'drugX' 'drugX']40     drugY51     drugX139    drugX197    drugX170    drugXName: Drug, dtype: object
Let’s check the accuracy of the model.
from sklearn import metricsimport matplotlib.pyplot as pltprint("DecisionTrees's Accuracy: ", metrics.accuracy_score(y_test,pred))
Out:
DecisionTrees's Accuracy:  0.9833333333333333
Create a Decision Tree and Visualize it.
from sklearn import treeplt.figure(figsize=(25,10))tree.plot_tree(Tree,filled=True,               rounded=True,               fontsize=14);
The actual code can be checked out on: