Decision Trees: Parametric Optimization | by Baban Deep Singh | Towards Data Science

As we begin working with data, we (generally always) observe that there are few errors in the data, like missing values, outliers, no proper formatting, etc. In nutshell, we call them inconsistency. This consistency, more or less, skews the data and hamper the Machine learning algorithms to predict correctly.
Previously we observed that how outliers can skew our analysis. Additionally, We observed that the k-NN classifier increased the accuracy once we removed the outliers and optimized its parameters, whereas for us our decision tree classifier performed badly. One speculation is that we did not optimize the parameters the classifier takes, so in this article, we will see if the classifier is not appropriate for this task or needed more consideration.
If you haven’t read the article you can find it here:
medium.com
Let's see if we can work with the parameters A DT classifier takes to uplift our accuracy.
class sklearn.tree.DecisionTreeClassifier(*, criterion='gini', splitter='best', max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features=None, random_state=None, max_leaf_nodes=None, min_impurity_decrease=0.0, min_impurity_split=None, class_weight=None, presort='deprecated', ccp_alpha=0.0)
The function to measure the quality of a split. There are 2 most prominent criteria are {‘Gini’, ‘Entropy’}.
The Gini Index is calculated by subtracting the sum of the squared probabilities of each class from one. It favors larger partitions.
Entropy or Information Gain multiplies the probability of the class times the log (base=2) of that class probability.
How about we use both of these criterion and check their interaction with max_depth, minimum_samples_split, minimum_weight_fraction_leaf.
The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node. Samples have equal weight when sample_weight is not provided.
fractions = [0,0.1,0.2,0.3,0.4,0.5]from sklearn.metrics import accuracy_scoremin_weight_fraction_leaf = []acc_gini = []acc_entropy = []for i in fractions: dtree = DecisionTreeClassifier(criterion='gini', min_weight_fraction_leaf=i ) dtree.fit(X_train, y_train) pred = dtree.predict(X_test) acc_gini.append(accuracy_score(y_test, pred)) #### dtree = DecisionTreeClassifier(criterion='entropy',min_weight_fraction_leaf=i) dtree.fit(X_train, y_train) pred = dtree.predict(X_test) acc_entropy.append(accuracy_score(y_test, pred)) #### min_weight_fraction_leaf.append(i)d = pd.DataFrame({'acc_gini':pd.Series(acc_gini),  'acc_entropy':pd.Series(acc_entropy), 'min_weight_fraction_leaf':pd.Series(max_depth)})# visualizing changes in parametersplt.plot('min_weight_fraction_leaf','acc_gini', data=d, label='gini')plt.plot('min_weight_fraction_leaf','acc_entropy', data=d, label='entropy')plt.xlabel('min_weight_fraction_leaf')plt.ylabel('accuracy')plt.legend()
From the graph below, we can observe that the accuracy is higher at 0.3 for both the criteria.
Why ?
After providing weights to the samples in the leaf nodes, terminating nodes, we are getting similar accuracy, possibly similar information gain and impurities shared between both criteria. Which implies that we are getting pure nodes.
The minimum number of samples required to split an internal node:
If int, then consider min_samples_split as the minimum number.
If float, then min_samples_split is a fraction and ceil(min_samples_split * n_samples) are the minimum number of samples for each split.
min_samples_split = []acc_gini = []acc_entropy = []for i in range(2,20,1): dtree = DecisionTreeClassifier(criterion='gini', min_samples_split=i ) dtree.fit(X_train, y_train) pred = dtree.predict(X_test) acc_gini.append(accuracy_score(y_test, pred)) #### dtree = DecisionTreeClassifier(criterion='entropy',min_samples_split=i) dtree.fit(X_train, y_train) pred = dtree.predict(X_test) acc_entropy.append(accuracy_score(y_test, pred)) #### min_samples_split.append(i)d = pd.DataFrame({'acc_gini':pd.Series(acc_gini),  'acc_entropy':pd.Series(acc_entropy), 'min_samples_split':pd.Series(max_depth)})# visualizing changes in parametersplt.plot('min_samples_split','acc_gini', data=d, label='gini')plt.plot('min_samples_split','acc_entropy', data=d, label='entropy')plt.xlabel('min_samples_split')plt.ylabel('accuracy')plt.legend()
From the above graph, we can see that entropy is working better at min_samples_split at 7.
Why?
Mathematically speaking, Entropy takes into account the samples, and their information gain to build up a tree. So here at the optimal value of No. of samples, we are getting more information gain regarding the further splits. On the other hand, Although gini index works best in bigger partitions. The data we have contains impurities which cause the accuracy decrease with increase in the min_samples_split.
The maximum depth of the tree. If None, then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples. So it is a good practice to have both the parameter work in harmony.
max_depth = []acc_gini = []acc_entropy = []for i in range(1,30): dtree = DecisionTreeClassifier(criterion='gini',max_depth=i ) dtree.fit(X_train, y_train) pred = dtree.predict(X_test) acc_gini.append(accuracy_score(y_test, pred)) #### dtree = DecisionTreeClassifier(criterion='entropy', max_depth=i) dtree.fit(X_train, y_train) pred = dtree.predict(X_test) acc_entropy.append(accuracy_score(y_test, pred)) #### max_depth.append(i)d = pd.DataFrame({'acc_gini':pd.Series(acc_gini),  'acc_entropy':pd.Series(acc_entropy), 'max_depth':pd.Series(max_depth)})# visualizing changes in parametersplt.plot('max_depth','acc_gini', data=d, label='gini')plt.plot('max_depth','acc_entropy', data=d, label='entropy')plt.xlabel('max_depth')plt.ylabel('accuracy')plt.legend()
From this, we can clearly see that Gini index outperformed entropy.
Why?
As we know Gini-Index takes into account the probability of the classes, therefore, The degree of Gini index varies between 0 and 1, where 0 denotes that all elements belong to a certain class or if there exists only one class, and 1 denotes that the elements are randomly distributed across various classes. A Gini Index of 0.5 denotes equally distributed elements into some classes.
In our example, we can conclude that at max_depth = 6, we observe the low impurity nodes thus increasing the accuracy of our model.
Revisiting our discussion from above we can draw the following conclusions:
Gini-Index is providing us with the highest accuracy with max depth = 6.
Entropy and Gini-index can behave similarly with appropriately selected min_weight_fraction_leaf.
With min_samples_split as 7, Entropy is outperforming Gini for a rudimentary assumption that More samples will provide more information gain and tend to skew the Gini index as the impurity increases.
Therefore with taking the criteria as Gini and max_depth = 6, we obtained the accuracy as 32% which is an 18% increase from without using parametric optimization. Hence, Optimizing the parameter rightfully, will increase the model accuracy and provide better results.