A Practical Approach to Supervised Learning
By: Shubhanker Singh
Tags: Machine Learning, Python, Supervised Learning, Classification, Data Science

Machine Learning is the art of teaching machines to make decisions from the data. There are multiple algorithms that help computers to analyze data and get valuable insight, and thus assisting it to make the decision on new data sets that it hasn’t seen before. Most of these algorithms fall into one of these three categories:

Supervised Learning: Algorithms that learn from labeled data, and do predictions on data never seen before. Unsupervised Learning: Algorithms that tries to find patterns and similarity in the unlabeled data so that it can be clustered accordingly. Reinforcement Learning: Algorithms that are allowed to interact with an environment and their performance is optimized by a system of reward and punishment.

In this article, we will be focussing on the Supervised Learning (SL) method. As stated earlier, SL uses labeled data and gives its predictions on unlabeled data. A labeled data is one which has been categorized in one or more category or has been given a particular value. For e.g., there is data of all the students of different schools taking part in an interschool competition, then, in this case, the school name could be used as a label to categorize the students. In another case, if there are multiple houses with different areas and they cost according to their areas, then the cost could be used as a label in this case. Although in both the instances the data was labeled, they were quite different in their label type. In the first example, the number of labels was discrete, whereas, in the second one, the label was some decimal values and thus continuous in nature. The SL is further categorized on this basis:

Classification: The value to be predicted is categorical and discrete. Regression: The value to be predicted is continuous in nature.

Classification

There are many cases when you would be using classification methods to make a prediction on the categorical data. Various examples include categorizing emails as spam or not spam, whether the cancer is malignant or benign, assigning plants or animals into a kingdom and species, etc. There are many different algorithms used in classification problems. Some of them are:

Logistic Regression Support Vector Machine (SVM) k-Nearest Neighbors (k-NN) Decision Tree

Here, I would only be demonstrating k-NN but would be writing about the others in the upcoming blogs. Also, I would be using the scikit-learn Iris dataset (Fisher, UC Irvine) that contains three classes with fifty instances each.

k-Nearest Neighbors Classifier

It is one of the simplest and super easy to learn classification algorithm. The main idea behind a k-NN algorithm is that the elements that are close to each other would belong to the same category. It may or may not be true for all the data points. It predicts the label of a data point by looking at the ‘k’ closest labeled data points. The unlabeled data point will be classified into that category which is a majority among the ‘k’ closest data points:

k-NN Classification for k = 3 and k = 5

In the diagram presented above, you can see that the predictions made by the k-NN classifier for two different values of ‘k’. In both cases, the classifier made different predictions. This might cause a stir in your mind about the value of ‘k’, but don’t worry I will talk about it later. Now since you have clear intuition about this algorithm, let’s implement on the Iris dataset and see how it works.

First, let me give you some insight into the Iris dataset. It is a very simple dataset that contains flower data. It contains four features namely petal length, petal width, sepal length, and sepal width. It also contains the target variable which are the flower categories namely Versicolor, Virginica, and Setosa. Each of these three labels has fifty instances each.

First and foremost, it is a good practice to import all the libraries you may need later.

from sklearn import datasets #importing datasets from sklearn

import pandas as pd #importing pandas with an alias pd

import numpy as np #importing numpy with an alias np

import matplotlib #impoting matplotlib

import matplotlib.pyplot as plt #importing pyplot with an alias plt

Notice that I imported the datasets module from sklearn . That would help in loading the Iris dataset. Now, it would be a good idea to do am exploratory data analysis on the given data. In order to do so, you will have to first load the Iris dataset and assign it to some variable, iris in this case:

iris = datasets.load_iris()

This data is in the form of a Bunch , which you can check using type(iris) . Bunch is similar to the dictionary as it also contains key-value pairs. Let’s check the keys of this dataset, which can be done using print(iris.keys()) , and it outputs dict_keys([‘data’, ‘target’, ‘target_names’, ‘DESCR’, ‘feature_names’, ‘filename’]) . The data contains the data for all the features namely petal length, petal width, sepal length, and sepal width, target contains the value of the target in numeric form (i.e., 0 for Setosa, 1 for Versicolor and 2 for Virginica) , target_names contain the name of the target variable (i.e., Setosa, Versicolor, Virginica), DESCR contains the description of this dataset about its contributor, statistics and many more, feature_names contain the name of the features (i.e., sepal length (cm), sepal width (cm), petal length (cm), petal width (cm)) and finally, filename contains the location of the file where it has been loaded.

Lets extract the data and target from iris by assigning it to some variable X and Y respectively.

X = iris['data']

Y = iris['target']

To perform further operations on the data, you should convert the data into a pandas dataframe and assign it to some variable say df using :

df = pd.DataFrame(X, columns = iris.feature_names) .

Doing a visual exploratory data analysis of Iris dataset using pd.plotting.scatter_matrix(df, c = Y, figsize = [15, 10],s=150) will give an output:

As you can see, the diagonal consist of histograms of the features corresponding to the row and the columns, and the non-diagonal plots are the scatter plots of the column features and the row features colored by their target variable. It is quite obvious to see a correlation between the value of the features and the target variables. Let’s plot a scatter plot of the petal length and the petal width individually and see this correlation clearly. It can be plotted using

plt.scatter(df[‘petal length (cm)’], df[‘petal width (cm)’], c = Y) :

Here the correlation gets even clear. The violet plot corresponds to Setosa, the blue plot corresponds to Versicolor and the yellow plot corresponds to the Virginica.

Before training a model on our dataset, it is very important to split it into a training set, validation set, and test set. Here, I will be splitting my data in a training set (70% of the data) and the test set (30% of the data). Scikit-learn helps us to do this very easily using its train_test_split module. To do so, you will have to first import it from sklearn.model_selection using from sklean.model_selection import train_test_split . It returns four arrays: the training data, the test data, the training labels, and the test labels. We unpack these in four variables, X_train, X_test, y_train, y_test in this case:

from sklean.model_selection import train_test_split

X_train, X_test, Y_train, Y_test = train_test_split(X, Y,

test_size = 0.3, random_state = 21,stratify = Y)

The test_size argument decides the percentage of data to be assigned for the test set, random_state sets seed for a random number generators that splits the data in train and test (setting the same seed every time will produce the same split), startify is set to the array containing the labels so that the labels are distributed in the train and test set as they are in the original dataset.

Finally, it’s time to implement the classifier, but first, we need to import it using from sklearn.neighbors import KNeighborsClassifier and then instantiate the classifier by setting the number of neighbors using knn = kNeighborsClassifier(n_neighbors = 6) . Here, I started with the number of neighbors equal to 6 and assigned it to a variable knn . To train the model, scikit-learn provides fit() method as we are to trying to fit the data to our classifier, and then to make a prediction on a new unlabeled data, it provides predict() method. We train the model on our training set produced using the train_test_split , and later will do the prediction on the test set.

from sklearn.neighbors import KNeighborsClassifier

knn = kNeighborsClassifier(n_neighbors = 6)

knn.fit(X_train, Y_train) #training on the X_train, Y_train

knn.predict(X_test) #testing on the X_test

This will output an array of the predicted label for the test set:

array([2, 1, 2, 2, 1, 0, 1, 0, 0, 1, 0, 2, 0, 2, 2, 0, 0, 0, 1, 0, 2, 2,2, 0, 1, 1, 1, 0, 0, 1, 2, 2, 0, 0, 1, 2, 2, 1, 1, 2, 1, 1, 0, 2,1])

To check the accuracy of our model we use score() method of k-NN on our test data and label, knn.score(X_test, Y_test) which gives 0.955555555556 . This is not a bad result for such a simple model.

The value of ‘k’ is a big deal in this classifier. The smaller value of ‘k’ means the model is very complex and can lead to overfitting (the model tries to fit all the points in the correct category), whereas, the larger value of ‘k’ means the model is less complex and has smoother decision boundary which may lead to underfitting (the model doesn’t fit most of the obvious points). There is a better value of ‘k’ which is neither big nor very small which doesn’t lead to overfitting or underfitting.

Regression

Regression is used when the target variable is a continuous value. A continuous value is one which is an integer, floating-point, etc. There are many examples of a regression problem including predicting house prices, predicting the stock values, etc. Since the value is continuous in the regression problem, its accuracy can not be evaluated. Thus it is evaluated by the value of a cost function which could be either Root Mean Square Error (RMSE) or Cross-Entropy Function or any other.

To demonstrate the regression I will be doing analysis on the Boston Housing dataset of Kaggle which can be downloaded from here. Scikit-learn also provides this dataset and can be used in a similar manner as we used the Iris dataset. I would be using it by downloading from Kaggle. If you want to see how it is loaded using the in-built scikit-learn dataset, you can do so here.

Now, after you have downloaded the dataset, you need to import Pandas and Numpy in your jupyter file. To load the data you can use pd.read_csv() and pass in the file (in the form of a string) as an argument:

import pandas as pd #importing pandas with an alias pd

import numpy as np #importing numpy with an alias np df = pd.read_csv('Boston.csv')

You can further check the data by using the head() method. It would by default show you the first five rows, but you can pass in the number of rows as an argument and can view as many records. It has 15 columns out of which the first column is for indexing, the next 13 columns are the features of the datasets, and the last column (i.e., medv) is the target variable which is the median value of the owner occupied home in thousands of dollars. If you wonder how do I know this, I simply checked the documentation of the dataset and you can do the same. As you can see the data loaded using pandas is combined of the feature and the target variables, but scikit-learn needs them in separate arrays. We can do so by splitting our dataset by dropping the medv column and using it as a target:

X = df.drop('medv', axis = 1).values #dropping the medv column

Y = df['medv'].values #using medv as target

We used the values attribute as it returns the NumPy array for us to use.

Linear Regression

Now it's the time to select our regression model that would help in predicting values for the unlabeled data. I would be choosing a very simple model called the Linear Regression (LR). LR defines an optimal line that could fit in all the give data and it assumes that all the data which it would accounter, later on, would also follow the same pattern. In one dimension (i.e., a dataset which has only one feature)it is a simple line with parameters a and b:

y = ax + b

Here, y is the target variable, x is the feature of the dataset, and a,b are the parameters that are to be learned. The best way to learn a,b is by defining a loss function and then minimizing it to get an optimal value of the parameters. Well, how to formulate a loss function over here? As you know, linear regression tries to fit the data on a line, but in the real case scenario, all the data may not fit on the line. The best that can be done is to minimize the vertical distance between the line and the data points. Bingo! Here lies the formula for our cost function. Don’t get confused by the term ‘cost’ as the loss function is also called the cost function or the error function. Now, this vertical distance is also known as ‘residual’. We can try to minimize the sum of the residual, but that may lead to cancellation of lot of positive residual with the negative ones, as you can see below:

To avoid that we minimize the sum of the squares of the residuals. This would be a perfect loss function and it is commonly known as Ordinary Least Squares (OLS). Scikit-learn performs this operation when we apply its Linear Regression model and try to fit our data to it.

This was the case when our data has one feature i.e., one-dimension data. For data with higher dimension, scikit-learn try to fit the data on this linear equation:

Linear Regression will have to learn about n+1 parameters.

Now that you know the logic behind the LR model let’s try to implement it on our Boston dataset. First, we will split our data in a training set and the test set using the train_test_split module of Scikit-learn as we did earlier in our classification example:

from sklearn.model_selection import train_test_split

X_train, X_test, Y_train, Y_test = train_test_split(X, Y,

test_size = 0.3, random_state = 21)

Then, we need to import the Linear Regression model from Scikit-learn using from sklearn.linear_model import LinearRegression and then instantiate it. Now, you can apply the fit() method on the training set and do the prediction on the test set using predict() method:

from sklearn.linear_model import LinearRegression

reg = LinearRegression()

reg.fit(X_train, Y_train)

pred = reg.predict(X_test)

Unlike classification, we cannot use accuracy to evaluate the regression model. In the case of a linear regression model, the performance is evaluated using R² , which is the evaluation of the amount of variance in the target variable that is predicted from the feature variable. To calculate R², we apply the score method and pass the arguments X_test and Y_test, reg.score(X_test, Y_test) .

Until now, I have been splitting the data in training set and the test set. But to make the model do an enhanced evaluation on a new dataset, we can use the technique called Cross-Validation (CV). Suppose we want to do n-fold cross-validation, we would split our data in n equal folds. Then, we will hold our first fold, fit our model on the remaining n-1, predict on the test set and compute the metric of interest. Next, we hold on our second set, fit on the remaining data, and compute the metric of interest. We continue to do so for all the n-folds.

We get n values of the metric of our interest (R² in our case). We can take the average of all of them or we can calculate other statistics such as mean, median, etc. However, a point should be noted that more is the number of the folds, more computationally expensive will be our model, as we are training and testing that many numbers of times. To implement LR with CV:

from sklearn.model_selection import cross_val_score

from sklearn.linear_model import LinearRegression

reg = LinearRegression()

cv = cross_val_score(reg, X, Y, cv = 5) #5-fold cross validation

print(cv)

It will output the R² value computed for all the five folds in the form of an array:

[ 0.57280576 0.72459569 0.59122862 0.08126754 -0.20963498]

Note that I used this linear regression only for the demonstration purpose and it is never used like this. What we generally use is a regularised linear regression model.

You can check the code for the classification model over here and the regression model over here.

I hope this tutorial helped you get started with Machine Learning. You should have a better idea about Classification and Regression Supervised Learning methods. I would be continuing this series and would be writing about Unsupervised learning in my upcoming blog.