Machine Learning Basics: Support Vector Machine (SVM) Classification | by Gurucharan M K | Towards Data Science

In the previous stories, I had given an explanation of the program for implementation of various Regression models. Also, I had described the implementation of the Logistic Regression and the KNN Classification model. In this article, we shall see the algorithm and the implementation of the SVM Classification with a crisp example.
The Support Vector Machine (SVM) Classification is similar to the SVR that I had explained in my previous story. In SVM, the line that is used to separate the classes is referred to as hyperplane. The data points on either side of the hyperplane that are closest to the hyperplane are called Support Vectors which is used to plot the boundary line.
In SVM Classification, the data can be either linear or non-linear. There are different kernels that can be set in an SVM Classifier. For a linear dataset, we can set the kernel as ‘linear’.
On the other hand, for a non-linear dataset, there are two kernels, namely ‘rbf’ and ‘polynomial’. In this, the data is mapped to a higher dimension which makes it easier to draw the hyperplane. Afterwards, it is brought down to the lower dimension.
From the above diagram, we can see that there are two classes of shapes, rectangle and circle. As it is difficult to draw a SVM line in the 2D Plane, we map the data points to a higher dimension (3D Plane) and then draw the hyperplane. It is then brought down to the original plane with the SVM Classifier drawn in red color.
In this way, the SVM Classifier can be used to classify a data point to which class it belongs from the given dataset. Let us use this algorithm to solve a real-world problem.
In this implementation of the SVM Classification model, we shall use a Social Network Advertisement dataset which consists of three columns. The first two columns are the independent variables, namely the ‘Age ’and the ‘EstimatedSalary’ and the last column is the dependent variable ‘Purchased’, which is in the binary format denoting whether the individual has bought the product (1) or not (0).
In this problem, we have to build a SVM Classification model for a company that will classify whether a user of a particular age and with a particular salary will buy their given product or not. Let us now go through the implementation of the model.
As always, the first step will always include importing the libraries which are the NumPy, Pandas and the Matplotlib.
import numpy as npimport matplotlib.pyplot as pltimport pandas as pd
In this step, we shall obtain the dataset from my github repositary which is stored as SocialNetworkAds.csv and store it to the variable dataset. Then we shall assign the corresponding variables to X and Y. Finally, we shall see the first 5 rows of our dataset.
dataset = pd.read_csv('https://raw.githubusercontent.com/mk-gurucharan/Classification/master/SocialNetworkAds.csv')X = dataset.iloc[:, [0, 1]].valuesy = dataset.iloc[:, 2].valuesdataset.head(5)>>Age   EstimatedSalary   Purchased19    19000             035    20000             026    43000             027    57000             019    76000             0
There are 400 rows in this dataset. We shall split the data into the training and the test set. In this the test_size=0.25 denotes that 25% of the data will be kept as the Test set and the remaining 75% will be used for training as the Training set. Thus, there are about 100 data points in the test set.
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25)
This step of feature scaling is an additional step that can increase the speed of the program as we scale down the values of X to a smaller range. In this, we scale down both the X_trainand the X_test to a small range of -2 to +2. For example, the salary 75000 is scaled down to 0.16418997.
from sklearn.preprocessing import StandardScalersc = StandardScaler()X_train = sc.fit_transform(X_train)X_test = sc.transform(X_test)
Once the training test is ready, we can import the SVM Classification Class and fit the training set to our model. The class SVC is assigined to the variable classifier. The kernel used here is the “rbf” kernel which stands for Radial Basis Function. There are several other kernels such as the linear and the Gaussian kernels which can also be implemented. The classifier.fit() function is then used to train the model.
from sklearn.svm import SVCclassifier = SVC(kernel = 'rbf', random_state = 0)classifier.fit(X_train, y_train)
In this step, the classifier.predict() function is used to predict the values for the Test set and the values are stored to the variable y_pred.
y_pred = classifier.predict(X_test) y_pred
This is a step that is mostly used in classification techniques. In this, we see the Accuracy of the trained model and plot the confusion matrix.
The confusion matrix is a table that is used to show the number of correct and incorrect predictions on a classification problem when the real values of the Test Set are known. It is of the format
The True values are the number of correct predictions made.
from sklearn.metrics import confusion_matrixcm = confusion_matrix(y_test, y_pred)from sklearn.metrics import accuracy_score print ("Accuracy : ", accuracy_score(y_test, y_pred))cm>>Accuracy :  0.9>>array([[59,  6],       [ 4, 31]])
From the above confusion matrix, we infer that, out of 100 test set data, 90 were correctly classified and 10 were incorrectly classified, leaving us with an accuracy of 90%.
In this step, a Pandas DataFrame is created to compare the classified values of both the original Test set (y_test) and the predicted results (y_pred).
df = pd.DataFrame({'Real Values':y_test, 'Predicted Values':y_pred})df>> Real Values   Predicted Values1             10             00             01             10             0... ...  ... ...1             11             10             00             01             1
This step is an additional step which is not much informative as the Confusion matrix and is mainly used in regression to check the accuracy of the predicted value.
In this last step, we visualize the results of the SVM Classification model on a graph that is plotted along with the two regions.
from matplotlib.colors import ListedColormapX_set, y_set = X_test, y_testX1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.01),                     np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.01))plt.contourf(X1, X2, classifier.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),             alpha = 0.75, cmap = ListedColormap(('red', 'green')))plt.xlim(X1.min(), X1.max())plt.ylim(X2.min(), X2.max())for i, j in enumerate(np.unique(y_set)):    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],                c = ListedColormap(('red', 'green'))(i), label = j)plt.title('SVM Classification')plt.xlabel('Age')plt.ylabel('EstimatedSalary')plt.legend()plt.show()
In this plot, There are two regions. The Red region denotes 0, which consists of people who have not bought the product and the Green region denotes 1, which consists of the people who have bought the product. As we have chosen a non-linear kernel (rbf), we are getting regions that are not separated by a linear line.
If you notice closely, we can see the 10 wrongly classified data points in the test set with the difference in color in the specific region.
Thus in this story, we have successfully been able to build a SVM Classification Model that is able to predict if a person is going to buy a product based on his age and salary. Do feel free to try out various other common classification datasets that are available on the net.
I am also attaching the link to my github repository where you can download this Google Colab notebook and the data files for your reference.
github.com
You can also find the explanation of the program for other Classification models below:
Logistic Regression
K-Nearest Neighbours (KNN) Classification
Support Vector Machine (SVM) Classification
Naive Bayes Classification (Coming Soon)
Random Forest Classification (Coming Soon)
We will come across the more complex models of Regression, Classification and Clustering in the upcoming articles. Till then, Happy Machine Learning!