Building and Extending Your Decision Tree: A Hands-On Guide
Source: tai_blog

Introduction   This post explores decision trees and guides how to build one from scratch. Ill start with the basics and gradually move on to more advanced techniques to enhance the decision tree.   Ill begin with a simple example to explain the basics of decision trees. Then  Ill delve into the mathematics behind them  covering key concepts like entropy and Gini impurity. Ill also introduce the soft trees using the logistic function.   After covering the theory  Ill dive into coding to show you how to build your decision tree  without using pre-built libraries. Finally  Ill explore advanced techniques to optimize the trees performance  such as using KS statistics and combining different metrics.   By the end of this guide  youll have a solid understanding of decision trees and the confidence to build and tweak your AI models. Lets get started!   Understanding the Basics with a Simple ExampleLets dive into the basics of decision trees using a simple example. Imagine we have data from 1000 individuals with different ages (our input variable x)  and we want to predict whether they are employed ( target variable Y  binary: 1 for employed  0 for not employed). The goal is to build a model f(x)=Y that predicts employment status.   To start  we need to find the best cut-off age that divides our data into two groups: those above the cut-off age and those below it. This split should maximize the difference in employment rates between the two groups.   For instance  lets assume that age 30 is the best cut-off point. This means the employment ratio for people older than 30 and those younger than 30 shows the largest difference. We can then create a simple decision rule: if a persons age is greater than 30  they are more likely to be employed  and if they are 30 or younger  they are less likely employed.   Mathematically  we can represent this decision rule as follows:   This is a step function where f(x) predicts 1 (employed) if the age x is greater than 30 and 0 (unemployed) if x is 30 or less. This simple model illustrates how a decision tree works by splitting the data at the optimal cut-off point to make predictions.   In reality  decision trees can handle multiple variables or a vector:   The tree considers different split points to find the best way to divide the data. The key aspects of building a decision tree are 1) choosing the best variable to split on  and 2) finding the optimal split point. Ill discuss these aspects in the next paragraph.   Mathematical Concepts and Extensions for Decision TreesUnderstanding Decision TreesLets dive into the math behind decision trees and investigate how they can be extended. Imagine a decision tree as a series of steps that divide the data into different regions  each associated with a different outcome. For instance  in our earlier example  we used age (x) to predict employment status (Y). We started with a simple decision rule that looked like this:   When extending this to multiple variables  the decision tree model can be expressed as:   Each decision node in the tree represents a condition or a set of conditions based on the features x_1  x_2    x_n. The model is built by recursively splitting the data according to these conditions. For example  a step function for two variables  age (x_1) and income (x_2)  might look like this:   For instance  if we have two variables  age (x_1) and income (x_2)  each with its respective cut point  the decision function would like this:   We would use the cut tree method to solve Y = F(x_1  x_2 ). Think of it like slicing a cake. We need to make the best cuts to ensure each piece has the right balance of ingredients.   Here is how it works: First  we identify the most significant variable for splitting. This decision is similar to choosing where to make the initial cut on a cake. Should the division be based on age  income  or another factor?   Impurity MeasuresThis involves selecting the best variable and the optimal split point based on criteria such as entropy  Gini impurity  and information gain.   Entropy is a measure of impurity or randomness in the data  calculated as:   Here S is the dataset  c is the number of classes  and p_i is the proportion of instances in class i. Imagine we split our data and make each group as pure as possible. Entropy measures the randomness or impurity. Lower entropy means a purer group.   Gini impurity is another measure of impurity given by:   This is another way to measure purity  calculating how often youd randomly pick the wrong item if you randomly picked from a group. Lower Gini impurity means fewer mistakes.   Information gain measures the reduction in entropy or impurity after a split:   This tells us how much weve improved the purity of the data after a split. It measures the reduction in entropy. A higher information gain means weve made a good split.   These metrics are important because they help the tree figure out the best places to make splits  ensuring that each split makes the groups as similar as possible. For example  if you predict whether people will buy a product based on age and income  you want to create groups where people are more likely to make similar decisions.   You might question how to use these impurity measures in practice. Dont worry; Ive got you covered. Heres a summary based on my experience and research:   When to Use Each Criterion?   Entropy: Think of entropy as the meticulous organizer. Use it when you want to be precise about how mixed your groups are. Its perfect for tasks where you care about the exact distribution of your classes.   Gini Impurity: Gini is your go-to for quick  reliable results. It gives you a good measure of purity without much fuss. when solving binary classification problems  its less heavy on the calculations and gets the job done efficiently.   Information Gain: Use this when using algorithms such as ID3  C4.5  or C5.0  since this measure is about understanding how much each split helps you. Its also useful for observing the added value of each decision in the decision tree.   Smooth Function for Soft TreesA natural question is: is there any smooth the-step function for the cut tree method?   This is important for neural networks since traditional decision trees are non-differentiable step functions  and struggle in training neural networks models. Using smooth soft tree-like functions in NN as activation functions will enable backpropagation and gradient descent. In addition  the smooth soft tree can facilitate training and improve performance by providing probabilistic outputs. The soft prevents overfitting by smoothing decision boundaries  creating a more stable and generalizable model.   To introduce the concept of a soft tree  we can use a logistic function to smooth the step function. The logistic function is defined as:   Where s is the cut point  B is a large positive value (penalty) defining the steepness. This function approximates a step function when B is large  as the above equation quickly approaches 0 for x<s and 1 for x>s. This allows us to assign probabilities to the tree branches  effectively transitioning from a soft to a hard decision tree.   Incorporating this into our tree  the prediction function can be updated as follows:   if soft is chosen  the prediction is based on the weighted average of p and the hard trees assignment   This approach introduces the concept of a soft tree  where the decision boundaries are smooth rather than sharp.   Relationship Between Impurity MeasuresLets examine how the soft trees function relates to the hard trees impurity measures. In logistic regression  the probability of the positive class is modeled using the logistic function:   Given this model  the likelihood function for a single data point (x  y) is:   The overall likelihood for the dataset is:   Taking the logarithm  we obtain the log-likelihood:   Substituting p from the logistic function:   This shows that maximizing the likelihood function in logistic regression aims to find the best coefficients A and B that separate the classes effectively. This is similar to finding the split that minimizes impurity in Gini or entropy terms.   We first compare the Logistic Regression Likelihood and Gini Impurity:   If the data is well-separated  the logistic regression models decision boundary will minimize overlap  resulting in pure subsets (low Gini impurity). As p approaches 0 or 1 (pure classes)  Gini impurity 2p(1p) approaches 0  indicating pure nodes.   We then compare the Logistic Regression Likelihood and Entropy:   The log-likelihood function has terms p log(p)  (1  p) log(1  p)  similar to the entropy formula. This means that by maximizing log-likelihood  we are also reducing entropy. We aim for confident classifications  where p is close to 0 or 1. When p is near 0 or 1  the entropy p log(p)  (1  p) log(1  p) becomes 0  indicating pure nodes.   Understanding and implementing soft tree methods with the logistic function is also important in deep learning and KolmogorovArnold Networks (KAN)  as it bridges the gap between decision trees and neural network architectures  enabling more flexible and powerful models.   In the next section  Ill discuss the code implementation for traditional decision trees. This can provide you with a clear understanding of their mechanics. In addition  it will help readers develop their trees and understand how the model works. Ill also show why a parametric approach with the logistic function is important.   Implementing a Traditional Decision Tree from ScratchI will study the code for building a decision tree classifier from scratch  without relying on pre-built libraries like scikit-learn. This hands-on approach will help you understand the underlying mechanisms of decision trees and how to implement them yourself. Lets dive into the code block by block  focusing on the key parts and more complex aspects of the tree growth and best split calculations.   Gini Impurity Calculation   def gini_impurity(y): m = len(y) return 1.0 - sum((np.sum(y == c) / m) ** 2 for c in np.unique(y))The function calculates the Gini impurity for a given set of labels y. Here  we use this measure in split-point searching.   Finding the Best Split   def best_split(X  y): best_gini = 1.0 best_idx  best_thr = None  None m  n = X.shape for idx in range(n): thresholds = np.unique(X[:  idx]) for thr in thresholds: left_mask = X[:  idx] < thr right_mask = ~left_mask if sum(left_mask) == 0 or sum(right_mask) == 0: continue gini = (sum(left_mask) * gini_impurity(y[left_mask]) + sum(right_mask) * gini_impurity(y[right_mask])) / m if gini < best_gini: best_gini = gini best_idx = idx best_thr = thr return best_idx  best_thrThe function finds the best feature and threshold to split the data by minimizing the Gini impurity. Here's a detailed breakdown of its process:   Initialization: best_gini is initialized to 1.0  representing the worst possible impurity. best_idx and best_thr will store the best feature index and threshold for splitting.Iterating Over Features and Thresholds: The function loops over each feature and each unique threshold value for that feature. For each combination:Masking: It creates masks to separate the data into left and right subsets based on whether the feature values are below or above the threshold.Gini Calculation: The Gini impurity for the split is calculated by weighting the Gini impurities of the left and right subsets by their sizes.Updating Best Split   If the current splits Gini impurity is lower than the best observed so far  the function updates best_gini  best_idx  and best_thr.   class Node: def __init__(self  gini  num_samples  num_samples_per_class  predicted_class): self.gini = gini self.num_samples = num_samples self.num_samples_per_class = num_samples_per_class self.predicted_class = predicted_class self.feature_index = 0 self.threshold = 0 self.left = None self.right = NoneThe class represents a node in the decision tree. It stores:   Gini: The Gini impurity of the node.num_samples: The number of samples at the node.num_samples_per_class: The distribution of samples per class.predicted_class: The class prediction at the node.feature_index and threshold: Used to store the best feature and threshold for splitting.left and right: Pointers to the left and right child nodes.Decision Tree Classifier   class DecisionTreeClassifier(BaseEstimator  ClassifierMixin): def __init__(self  max_depth=None): self.max_depth = max_depth def fit(self  X  y): # Check for NaN or infinite values in the data if np.any(np.isnan(X)) or np.any(np.isnan(y)): raise ValueError(Input data contains NaN values.) if np.any(np.isinf(X)) or np.any(np.isinf(y)): raise ValueError(Input data contains infinite values.) self.classes_ = np.unique(y) self.n_classes_ = len(self.classes_) self.n_features_ = X.shape[1] self.tree_ = self._grow_tree(X  y) return selfThe class is the main class for our decision tree. It inherits from BaseEstimator and ClassifierMixin to ensure compatibility with scikit-learn. Heres what each method does:   Initialization: Sets the maximum depth of the tree.Fitting the Model: Checks for NaN or infinite values in the input data  initializes class and feature information  and starts the tree-growing process.Growing the Tree   def _grow_tree(self  X  y  depth=0): num_samples_per_class = [np.sum(y == i) for i in range(self.n_classes_)] predicted_class = np.argmax(num_samples_per_class) node = Node( gini=gini_impurity(y)  num_samples=X.shape[0]  num_samples_per_class=num_samples_per_class  predicted_class=predicted_class  ) if depth < self.max_depth: idx  thr = best_split(X  y) if idx is not None: indices_left = X[:  idx] < thr X_left  y_left = X[indices_left]  y[indices_left] X_right  y_right = X[~indices_left]  y[~indices_left] node.feature_index = idx node.threshold = thr node.left = self._grow_tree(X_left  y_left  depth + 1) node.right = self._grow_tree(X_right  y_right  depth + 1) return nodeThe method recursively builds the decision tree:   Node Creation: Initializes a new node with the Gini impurity  sample counts  and predicted class.Splitting the Node: It finds the best split if the current depth is less than max_depth. If a valid split is found:Left and Right Subsets: Creates subsets of the data based on the split.Recursive Calls: Recursively grows the left and right subtrees.Making Predictions   def predict(self  X): return np.array([self._predict(inputs) for inputs in X]) def _predict(self  inputs): node = self.tree_ while node.left: if inputs[node.feature_index] < node.threshold: node = node.left else: node = node.right return node.predicted_classThe predict method uses the helper function _predict to traverse the tree from the root to a leaf node  making predictions for each input:   Traversing the Tree: Starting from the root node  it moves to the left or right child based on the feature value and threshold.Leaf Node: Once a leaf node is reached  it returns the predicted class stored at the leaf.This custom implementation of a decision tree classifier provides a comprehensive understanding of how decision trees work. With this knowledge  you can tackle more advanced applications and customizations in machine-learning algorithms.   Building My Decision Tree with Enhanced Splitting CriteriaIn this section  I will demonstrate an extended decision tree method that incorporates the Kolmogorov-Smirnov (KS) statistic  entropy  and a combination of both to determine the best split. Additionally  we will introduce a soft decision tree factor  using the logistic function:   to assign probabilities to branches  making the decision tree more adaptable and robust.   Why Use KS?The KS statistic measures the maximum difference between the cumulative distributions of two samples  so  in this paper  I would use it to identify the most significant split points.   Traditionally  the Kolmogorov-Smirnov (KS) statistic is used to test the hypothesis that two samples come from the same distribution. It measures the maximum distance between the two samples empirical cumulative distribution functions (CDFs). Mathematically  the KS statistic for two samples F_1(x) and F_2(x) is defined as:   This method has also proved effective in distinguishing between two classes. So I make it a useful tool for decision trees.   Key Parts of the CodeHeres a closer look at the critical sections of the code:   Initialization: The class is initialized with parameters like depth  minimum samples per split  number of potential cut points  split criterion  and whether to use the 'soft' tree option.class DecisionTree(BaseEstimator  ClassifierMixin): def __init__(self  depth=10  min_samples_split=2  num_cut_points=100  split_way='entropy'  soft=False  B=5): self.depth = depth self.min_samples_split = min_samples_split self.num_cut_points = num_cut_points self.split_way = split_way self.soft = soft self.B = B self.tree = NoneBuilding the Tree: The method constructs the tree recursively. It splits the data into left and right subsets based on the best-split point determined by the _best_split method. A leaf node is created if the maximum depth is reached or the number of samples is below the minimum.   def _build_tree(self  X  y  depth): if depth == 0 or len(X) < self.min_samples_split: return self._leaf_value(y) feat_idx  threshold = self._best_split(X  y) if feat_idx is None: return self._leaf_value(y) left_idx = X[:  feat_idx] < threshold right_idx = ~left_idx left_tree = self._build_tree(X[left_idx]  y[left_idx]  depth-1) right_tree = self._build_tree(X[right_idx]  y[right_idx]  depth-1) return (feat_idx  threshold  left_tree  right_tree)Finding the Best Split: The _best_split method evaluates potential split points using KS  entropy  or both. It iterates over thresholds for each feature and calculates the KS statistic and entropy between the left and right splits. The split with the highest score is chosen.   def _best_split(self  X  y): best_score = -np.inf split_idx  split_thresh = None  None for feat_idx in range(X.shape[1]): thresholds = np.linspace(np.min(X[:  feat_idx])  np.max(X[:  feat_idx])  self.num_cut_points) for threshold in thresholds: left_idx = X[:  feat_idx] < threshold right_idx = ~left_idx if sum(left_idx) == 0 or sum(right_idx) == 0: continue if self.split_way == 'KS': ks_stat  _ = ks_2samp(y[left_idx]  y[right_idx]) score = ks_stat elif self.split_way == 'entropy': left_entropy = entropy(np.bincount(y[left_idx]  minlength=self.n_classes_) + 1e-10) right_entropy = entropy(np.bincount(y[right_idx]  minlength=self.n_classes_) + 1e-10) total_entropy = (sum(left_idx) * left_entropy + sum(right_idx) * right_entropy) / len(y) info_gain = entropy(np.bincount(y  minlength=self.n_classes_)) - total_entropy score = info_gain elif self.split_way == 'both': ks_stat  _ = ks_2samp(y[left_idx]  y[right_idx]) left_entropy = entropy(np.bincount(y[left_idx]  minlength=self.n_classes_) + 1e-10) right_entropy = entropy(np.bincount(y[right_idx]  minlength=self.n_classes_) + 1e-10) total_entropy = (sum(left_idx) * left_entropy + sum(right_idx) * right_entropy) / len(y) info_gain = entropy(np.bincount(y  minlength=self.n_classes_)) - total_entropy score = (ks_stat + info_gain) / 2 else: raise ValueError(fUnknown split_way: {self.split_way}) if score > best_score: best_score = score split_idx = feat_idx split_thresh = threshold return split_idx  split_threshPredicting with the Soft Tree: The _predict_proba method uses a logistic function to assign probabilities to each branch. This makes the tree 'soft'  allowing it to handle uncertainty better.   def _predict_proba(self  inputs  tree): if not isinstance(tree  tuple): return np.eye(self.n_classes_)[tree] feat_idx  threshold  left_tree  right_tree = tree prob = 1 / (1 + np.exp((inputs[feat_idx] - threshold) * self.B)) left_prob = self._predict_proba(inputs  left_tree) right_prob = self._predict_proba(inputs  right_tree) combined_prob = prob * left_prob + (1 - prob) * right_prob combined_prob /= combined_prob.sum() return combined_probHere are the results:   The results show that the KS statistic can boost decision tree performance. While the soft tree doesnt improve performance  it helps understand how to integrate probabilistic assignments into decision trees. This approach is useful for neural networks  where smooth activation functions are important for backpropagation and gradient descent. The soft tree method adds flexibility and potential for further optimization in machine learning models.   ConclusionI wrote this guide since decision trees are important tools in machine learning methods like XGBoost  LightGBM  and neural networks. While people often use pre-built packages like scikit-learn  understanding the underlying mechanics and math of decision trees allows for improvements and new algorithm creation.   By building a decision tree from scratch  you learn how splits are decided  how impurity measures guide these splits  and how the KS statistic can improve performance. I also introduce the concept of soft trees  which is useful for integrating probabilities and is important for neural networks.   Mastering decision tree basics helps you fine-tune and optimize models for specific tasks. It is also helpful for building advanced applications and customizations in machine learning projects.   The Python scripts are available in my GitHub repository: GitHub  datalev001/decsiontree.