Advanced Decision Tree Questions
________________________________________
1.	Pruning and Overfitting:
A decision tree is trained on a dataset, and its performance on the training set and test set is as follows:
o	Training accuracy: 98%
o	Test accuracy: 75%
After pruning the tree, the new accuracies are:
o	Training accuracy: 85%
o	Test accuracy: 80%
a. Why did the test accuracy improve after pruning the tree, even though the training accuracy decreased?
b. Describe how the Minimum Description Length (MDL) principle could be applied to this scenario.
c. Suppose the validation set accuracy after pruning is 82%. Would you prefer this pruned tree, or the one before pruning?
________________________________________
2.	Feature Importance Using Gini Reduction:
Given a dataset with three features (Feature 1, Feature 2, Feature 3) and the target class with two classes (Class A, Class B), the decision tree uses Gini impurity as the splitting criterion. After the tree is trained, the following Gini reductions are observed:
o	Feature 1: Gini Reduction = 0.12
o	Feature 2: Gini Reduction = 0.25
o	Feature 3: Gini Reduction = 0.08
a. Which feature is the most important for the model based on Gini Reduction?
b. If Feature 2 is removed from the dataset, what impact would you expect on the model's accuracy, and why?
c. Suppose Feature 2 is highly correlated with Feature 1. How could this affect feature importance and decision tree performance?
________________________________________
3.	Cross-Validation and Hyperparameter Tuning:
You are using 5-fold cross-validation to tune the depth of a decision tree. The average cross-validation accuracy for different depths is as follows:
o	Depth = 3: Accuracy = 75%
o	Depth = 5: Accuracy = 82%
o	Depth = 7: Accuracy = 84%
o	Depth = 10: Accuracy = 80%
o	Depth = 15: Accuracy = 75%
a. Based on these results, what is the optimal depth of the tree? Explain why.
b. What might cause the drop in accuracy for deeper trees (e.g., depth = 10 and 15)?
c. How would you use early stopping criteria in the tree-growing process to avoid overfitting?
________________________________________
4.	Multi-Class Classification with Decision Trees:
You are building a decision tree for a classification problem with four classes: A, B, C, and D. The dataset has the following class distribution:
Class	Count
A	50
B	50
C	100
D	50
5.	a. Calculate the Gini impurity of the dataset before any splits.
b. If a split divides the dataset into two subsets with the following distributions:
o	Subset 1: 30 A, 20 B, 50 C, 0 D
o	Subset 2: 20 A, 30 B, 50 C, 50 D
Calculate the weighted Gini impurity after the split.
c. Is this split beneficial? Explain your answer based on the reduction in Gini impurity.
________________________________________
5.	Handling Continuous Variables:
A decision tree is trained on a dataset with a continuous feature XXX and binary target class. The feature XXX has the following sorted values:
X=[1.2,2.5,2.7,3.1,3.5,4.0,5.0,5.5,6.0,6.2]X = [1.2, 2.5, 2.7, 3.1, 3.5, 4.0, 5.0, 5.5, 6.0, 6.2]X=[1.2,2.5,2.7,3.1,3.5,4.0,5.0,5.5,6.0,6.2]
The corresponding target class labels are:
Y=[0,0,1,1,1,0,1,0,0,1]Y = [0, 0, 1, 1, 1, 0, 1, 0, 0, 1]Y=[0,0,1,1,1,0,1,0,0,1]
a. Identify all possible split points for the continuous feature XXX.
b. Suppose we split at X=3.5X = 3.5X=3.5. Calculate the Gini impurity for the two resulting subsets.
c. How does the choice of the split point affect the decision tree’s ability to handle continuous variables?
________________________________________
6.	Handling Imbalanced Data:
You are building a decision tree on an imbalanced dataset where 90% of the samples belong to Class A and 10% to Class B. The dataset is split as follows:
o	Subset 1: 80 samples from Class A, 5 samples from Class B.
o	Subset 2: 10 samples from Class A, 5 samples from Class B.
a. Calculate the Gini impurity for the overall dataset before any splits.
b. Calculate the weighted Gini impurity for each subset after the split.
c. How would you handle this class imbalance problem while building the decision tree? Suggest techniques like cost-sensitive learning or resampling.
________________________________________
7.	Effect of Noisy Data on Decision Trees:
A decision tree is trained on a noisy dataset where 5% of the class labels are randomly flipped. The tree has a depth of 20 and 512 leaf nodes. After training, the tree achieves 100% accuracy on the training data but only 65% on the test set.
a. Why does the decision tree achieve perfect accuracy on the training data despite the noise?
b. How would pruning help in this situation?
c. How can you modify the training process or the dataset to reduce overfitting caused by noisy labels?