Decision Tree Algorithm In Machine Learning
By: Bhanwar Saini
Tags: Python3, Programming, Artificial Intelligence, Data Science, Machine Learning

A decision tree is a non-parametric supervised machine learning algorithm. It is extremely useful in classifying or labels the object. It works for both categorical and continuous datasets. It is like a tree structure in which the root node and its child node should be present. It has a child node that denotes a feature of the dataset. Prediction can be made with a leaf or terminal node.

Recursive Greedy Algorithm

A recursive greedy algorithm is a very simple, intuitive algorithm that is used in the optimization problems.

What a recursive greedy algorithm does that at every step you have a choice. Instead of evaluating all choices recursively and picking the best one choice, go with that, Recurse and do the same thing. So basically a recursive greedy algorithm picks the locally optimal choice hoping to get the best globally optimal solution.

Greedy algorithms are very powerful in some problems, such as Huffman encoding, or Dijkstra’s algorithm, which is used in data structure and algorithms. We will be using this algorithm for the formation of the tree.

Step for learning the decision tree:

step 1: Start with an empty tree

step 2: Select a feature to split the data

For each split of the tree :

step 3: If nothing to do more, predict with the last leaf node or terminal node

step 4: Otherwise, go to step 2 & continue (recurse) to split

decision tree example

For example — let’s say, start with an empty tree and I pick a feature to split on. In our case, we split on credit. So we decided to say, take my data and split on which data points have excellent credit, which ones have fair credit and which ones have poor credit, and then for each subset of data, excellent, fair, poor, I continue thinking about what to do next. So I decide in the case of excellent credit, there was nothing else to do. So I stop, but in the other two cases, there was more to do and what I do is what’s called recursion. I go back to step two but only look at the subset of the data that has credit fair and then only look at a subset of data that has credit poor. Now if you look at this algorithm so far, it sounds a little abstract, but there are a few points that we need to make more concrete. We have to decide how to pick the feature to split on. We split on credit in our example, but we could have split on something else like the term of the loan or my income. And then since we having recursion here at the end, we have to figure out when to stop recursion, when to not go, and expand another node in the tree.

Problem 1: Feature split selection

Given a subset of dataset m(a node in a tree)

For each feature h(x):

Split data of M according to feature h (x) Compute classification error split

Chose feature h*(x) with lowest classification error

classification error is the number of mistakes in node and divides by the number of the data points in that node. For example — In the above diagram in the root node we made 18 as a risky loan that would be considered as a mistake and the total number of the data points in the root node will 40 so we can calculate the classification error of the root node. The second thing we will calculate the classification error of other nodes and try to see which feature has the lowest error that would our best split node.

Problem 2: Where will we do stopping the condition?

The first stop in the condition is stopped splitting when all the data agree on the value of y.

when already split on all the features. we can say there is nothing left in our dataset.

The Common Parameters of the Decision Tree

criterion: Gini or entropy, (default=gini)

The function to measure the quality of a split. Supported criteria are Gini for the Gini impurity and “entropy” for the information gain. It decides which node to splitting starts.

max_depth: int or None, (default = None)

The first hyperparameter is to tune in a decision tree is max_depth.

max_depth is what name suggests the maximum depth of the tree that you allow the tree to grow to

The deeper the tree, the more splits it has & it captures more information about the data.

however, in general, a decision tree overfits for large depth values. The tree perfectly predicts all of the train data, but it fails to capture the pattern in new data.

so you have to find the right max depth using hyperparameter tuning either grid search or random search to arrive at the best possible values of max_depth.

min_samples_split: int or float, (default = 2)

An internal node will have further splits (also called children)

min_samples_split specifies the minimum number of sample required to split an internal node.

we can either specify a number to denote the minimum number or a fraction to denote the percentage of samples in an internal node.

min_samples_leaf: int or float (default = 1)

A leaf node is a node without any children(without any further splits).

min_samples_leaf is the minimum number of samples required to be at a leaf node.

This parameter is similar to min_sample_splits, however, this describes the minimum number of samples at the leaf, the base of the tree.

This hyperparameter can also avoid overfitting.

max_features: int, float, string (default = None)

max_features represents the number of features to consider when looking for the best split.

We can either specify a number to denote the max_features at each split or a fraction to denote the percentage of features to consider while making a split.

we also have options such as sqrt, log2, None.

This method is used to control overfitting. In fact, it is similar to the technique used in a random forest, except in random forest we start with sampling also from the data and we generate multiple trees.

Code For Decision Tree Algorithms:

We will be using a dataset from the LendingClub. A parsed and clean form of the dataset is available here. Make sure you download the dataset before running the following command.

The train and validation dataset can found here.

Thanks for reading…..

Recommended Article