1. Introduction
The objective of this assignment is to demonstrate the implementation of decision trees for
classification and regression problems. This document freely reproduces some examples
shown in the excellent scikit-learn documentation.
2. Decision Trees
Decision trees are nonparametric learning methods used for classification and regression
problems. The goal is to create a model that predicts the values of the target variable, based
on a set of sequences of decision rules inferred from the training data. The tree therefore
approximates the target by a succession of if-then-else rules. This paradigm applies to both
categorical and numerical data. The more complex the tree generated, the better the model
“explains” the learning data but the higher the risk of over-fitting.
Decision trees have several advantages that make them interesting in contexts where it is
useful to understand the sequence of decisions made by the model:
• They are easy to understand and visualize.
• They require little data preparation (normalization, etc.).
• The cost of using trees is logarithmic.
• They can use categorical and numerical data.
• They can deal with multi-class problems.
• White box model: the result is easy to conceptualize and visualize.
However, these models have two major disadvantages:
2
• Over-fitting: sometimes the generated trees are too complex and generalize badly.
Choosing good values for the maximum depth (max_depth) and minimum number
of samples per leaf (min_samples_leaf) parameters avoids this problem.
• It may happen that the generated trees are not balanced, which implies that the
travel time is no longer logarithmic. It is therefore recommended to adjust the
dataset before construction, to avoid that one class largely dominates the others in
terms of the number of training examples.
3. Trees for classification
In scikit-learn, the class sklearn.tree.DecisionTreeClassifier allows to perform a multi-class
classification using a decision tree.
We start by importing the right modules and building the tree object:
from sklearn import tree
clf = tree.DecisionTreeClassifier()
For the example, we can define a minimalist dataset (two points, each in a class):
X = [[0, 0], [1, 1]]
y = [0, 1]
The tree is built using the .fit(X, y) method:
clf = clf.fit(X, y)
Prediction on new samples is done in the usual way with .predict(X):
clf.predict([[2., 2.]])
One can also predict the probability of each class for a sample:
clf.predict_proba([[2., 2.]])
4. Classification of Iris data
DecisionTreeClassifier is able to handle multi-class classification problems (eg with labels
0, 1, … K-1). In this example we will work with the Iris dataset, easily accessible in sklearn.
This dataset contains 150 instances of iris (a type of plant, each observation describes its 
3
morphology). The objective is to classify each instance into one of three categories: Iris
setosa, Iris virginica or Iris versicolor.
One of the classes is linearly separable with respect to the other two, but the other two are
not separable with respect to each other.
The attributes of the dataset are:
• sepal length,
• sepal width,
• petal length,
• petal width,
• class: Iris Setosa, Iris Versicolor or Iris Virginica.
The Iris dataset being very common, scikit-learn offers a native function to load it into
memory:
from sklearn.datasets import load_iris
iris = load_iris()
X, y = iris.data, iris.target
Question 1: Calculate the statistics (mean and standard deviation) of the four explanatory
variables: sepal length, sepal width, petal length and petal width.
Question 2: How many examples of each class are there?