ML: Decision Trees & Random Forests
Tags: Decision Tree, Machine Learning, Random Forest, Data Science

Decision trees are widely used in many fields but it is very poorly understood at the same time because we cannot certify our solution is the best or close to the best solution. However, the idea is really simple. It just follows the human decision process. It is asking a series of queries and decide the result.

It makes partition the input space based on the queries. We can divide the target classes with a rectangle region in the graph. However, it is a very low likelihood in a real-life that it follows the rectangle area but we assume the decision boundary is rectangle in decision trees. The rectangle is generated in a repeated manner.

The biggest advantage of the decision tree is interpretability. You don’t want to explain deep learning to your boss but the boss will understand the decision tree more easily.

Regression Trees

You can intuitively understand the decision trees can be used for classification. You can actually use it for regression. It has the same intuition as KNN. It fits a constant in each region.

You can express regression trees like this. R is the region. c is the assigned constant. The region is decided by decision trees. You don’t have to use the constant. We can use linear regression in the specific region or other regression methods if you have enough points falling in the region. We call this application as model trees.

Now, we need to know how we assign the constant and how we divide the region. How many regions are needed is not given apriori. We need to find out how many regions are needed during fitting the model. Our objective function is the mean squared error. We need to minimize the objective function. What is the reasonable guess for the constant? It is obviously the average of the points in that region.

Finding the best region is an np-complete problem. Therefore, we need to use heuristics. We will use the greedy algorithm. We need to define a splitting variable and splitting point.

j and s are a splitting variable and a splitting point each other. We find the j and s with this function.

c is calculated by averaging and the evaluation of function is determined by j and s. We find the j and s which lead this function to the minimum value. We find j by selecting each j and we don’t do this process jointly. Then, it calculates which s is the best. How do we find s? We sort the data and hopping on the data point like recursively and compare the results. The splitting is the same in a specific range. Thus, we are just hopping on the data.

When do we stop?

One strategy is early stopping. If the errors are really small, then we stop the algorithm. However, it can go wrong when the data point has interaction. Thus, it is not a good strategy. We call these effects as interaction effects.

We can stop when the data points in the leaves are small. We can choose the number of data points in the leaves when you start training.

Pruning

You build the perfect tree using a greedy algorithm. Then we prune the tree to get a smaller tree that is almost the same with the perfect tree with respect to accuracy.

Reduced Error Pruning

We need two sets of data, a training set, and a validation set. We train the big tree with the training set. We test it on the validation set. The pruning is performed on the validation set. We record the score of the big tree and replace the internal node to leave like the above graph. We test the pruned tree on the validation set. The score could go up or go down. If the score goes up or stays, then we keep the pruned tree. We do this until our algorithms are not improving. It can give a very different tree depending on the randomness.

Cost Complexity Pruning

We build the full tree and collapse the leaves. We store every collapsed tree fro a full tree to a single region tree. Each of the trees is subtrees of the full tree. We calculate the score on every tree and plus the complexity term in the equation.

|T| will be the number of leaves, it represents the complexity. We pick the smallest Ca tree. We can control the tree size by alpha. Alpha means how much we consider the tree size. We can use cross-validation for alpha.

Classification

Miss Classification Error

It is the basic probability function of the point in region m. We don’t use the sum squared error function. We are trying to maximize the probability and we need to define the cost function. It will be the classification error function.

Finding regions is the same as regression.

Gini Index

Gini index

It was invented by economists to observe the disparity of wealth. If there is more class 1 than class 2 in the specific region, we can use the Gini index representing the skewness. We are going to make region skewed distribution.

The number of bits in the information theory can be interpreted as the requirement of the quantity of information to explain the variable. Therefore, if you can gain the information by splitting, then it was nice splitting. For example:

<400, 400> => <400, 150>, <0, 250>. In this case, we can explain the split data with less information because we gained the information by splitting.

We minimize the cross-entropy or maximize the information gain. This is how we split the nodes.

Note: Miss Classification Error is used for final scoring.

Random Forests

Bagging and Ensemble

Random Forests are the result of a combination of bagging and ensemble. Bagging short for bootstrap aggregating. It uses the same training algorithm for every predictor and trains them on different random subsets of the training sets. When sampling is performed with replacement, this method is called bagging. When sampling is performed without replacement, this method is called pasting. After we trained all predictors, we aggregate the prediction. The typical aggregation function is the statistical mode for classification or the average for regression.

Out of Bag Evaluation

If you are doing bagging, then you can realize there is a subset that is not used in the bagging. Let’s say we sample 63% of data with replacement and train 5 models. Then, we can calculate the estimation of unused samples with a simple equation, approximately 0.7% of data is not used. We can use this unused data as the test set and this method is called Out of Bag Evaluation. It is frequently used because it produces similar results.

Feature Importance

Random Forests provide the feature importance calculation because you can easily calculate which feature reduce the impurity in the nodes. There are two methods. The first one is to calculate the impurity reduction in each feature of all trees and add the results and average them. How much Gini index increases or how much cross-entropy is decreased after this feature. The second way is to calculate the OOB error over all trees containing the specific variables. We re-measure the OOB error after we permute the values of the specific variable. Compute the differences and rank them.

Note: the root of p is the default value of 0the number of variables per tree and the default value of minimum node size is 1 => sci-kit learn.

Note: You should consider whether your data set contains relevant data or not because RF can give you very poor performance if you don’t have relevant data in the high dimensional dataset.

Notes: RF can be applied in Proximity of data points, Missing value imputation, Outlier detection, and Class Prototypes. You can also apply it in unsupervised learning but it is not common.

This is published on 9/26/2020