Unlocking the True Power of Support Vector Regression | by Ashwin Raj | Towards Data Science

SVMs or Support Vector Machines are one of the most popular and widely used algorithm for dealing with classification problems in machine learning. However, the use of SVMs in regression is not very well documented. This algorithm acknowledges the presence of non-linearity in the data and provides a proficient prediction model.
In this article, I will first try to give you an intuitive understanding of the algorithm by taking a deep-dive into the theory behind the algorithm. Then we will build our very own SVM Regressor model. And finally, we will look into some advantages of using Support Vector Regression.
The SVM regression algorithm is referred to as Support Vector Regression or SVR. Before getting started with the algorithm, it is necessary that we have an intuition of what a support vector machine actually is.
In machine learning, Support Vector Machines are supervised learning models with associated learning algorithms that analyze data used for classification and regression analysis. In Support Vector Regression, the straight line that is required to fit the data is referred to as hyperplane.
The objective of a support vector machine algorithm is to find a hyperplane in an n-dimensional space that distinctly classifies the data points. The data points on either side of the hyperplane that are closest to the hyperplane are called Support Vectors. These influence the position and orientation of the hyperplane and thus help build the SVM.
Now that we have an intuition of what a support vector machine is, we will take look into the various hyperparameters that are used in Support Vector Regression. Some of the key parameters used are as mentioned below:
Hyperplanes are decision boundaries that is used to predict the continuous output. The data points on either side of the hyperplane that are closest to the hyperplane are called Support Vectors. These are used to plot the required line that shows the predicted output of the algorithm.
A kernel is a set of mathematical functions that takes data as input and transform it into the required form. These are generally used for finding a hyperplane in the higher dimensional space.
The most widely used kernels include Linear, Non-Linear, Polynomial, Radial Basis Function (RBF) and Sigmoid. By default, RBF is used as the kernel. Each of these kernels are used depending on the dataset.
These are the two lines that are drawn around the hyperplane at a distance of ε (epsilon). It is used to create a margin between the data points.
Support Vector Regression is a supervised learning algorithm that is used to predict discrete values. Support Vector Regression uses the same principle as the SVMs. The basic idea behind SVR is to find the best fit line. In SVR, the best fit line is the hyperplane that has the maximum number of points.
Unlike other Regression models that try to minimize the error between the real and predicted value, the SVR tries to fit the best line within a threshold value. The threshold value is the distance between the hyperplane and boundary line. The fit time complexity of SVR is more than quadratic with the number of samples which makes it hard to scale to datasets with more than a couple of 10000 samples.
For large datasets, Linear SVR or SGD Regressor is used. Linear SVR provides a faster implementation than SVR but only considers the linear kernel. The model produced by Support Vector Regression depends only on a subset of the training data, because the cost function ignores samples whose prediction is close to their target.
Now that we have a gist of what Support Vector Regression is, we shall try to build our very own SVR regressor. The code and other resources for building this regression model can be found here.
Our first step is to import the libraries required to build our model. It is not necessary to import all the libraries at just one place. Python gives us the flexibility to import libraries at any place. To get started we will be importing the Pandas, Numpy, Matplotlib and Seaborn libraries.
#Import the Libraries and read the data into a Pandas DataFrameimport pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport seaborn as snstest = pd.read_csv("california_housing_test.csv")train = pd.read_csv("california_housing_train.csv")
Once these libraries have been imported our next step will be fetching the dataset and loading the data into our notebook. For this example I have used the California Housing dataset.
After successfully loading the data, our next step is to visualize this data. Seashore is an excellent library that can be used to visualize the data.
#Visualise the dataplt.figure()sns.heatmap(data.corr(), cmap='coolwarm')plt.show()sns.lmplot(x='median_income', y='median_house_value', data=train)sns.lmplot(x='housing_median_age', y='median_house_value', data=train)
Feature engineering is the process of using domain knowledge to extract features from raw data via data mining techniques. For this model I have selected columns with only numerical values. For handling categorical values label encoding techniques are applied.
#Select appropriate featuresdata = data[[‘total_rooms’, ‘total_bedrooms’, ‘housing_median_age’, ‘median_income’, ‘population’, ‘households’]]data.info()data['total_rooms'] = data['total_rooms'].fillna(data['total_rooms'].mean())data['total_bedrooms'] = data['total_bedrooms'].fillna(data['total_bedrooms'].mean()
Feature Scaling basically helps to normalize the data within a particular range. Normally several common class types contain the feature scaling function so that they make feature scaling automatically.
After selecting the desired parameters the next step is to import train_test_split from sklearn library which is used to split the dataset into training and testing data.
#Split the dataset into training and testing dataimport train_test_splitX_train, X_test, y_train, y_test = train_test_split(train, y, test_size = 0.2, random_state = 0)y_train = y_train.reshape(-1,1)y_test = y_test.reshape(-1,1)
After this SVR is imported from sklearn.svm and the model is fit over the training dataset.
# Fit the model over the training datafrom sklearn.svm import SVRregressor = SVR(kernel = 'rbf')regressor.fit(X_train, y_train)
Here, In this particular example I have used the RBF Kernel. Other parameters of the model are left to have their default configurations. Once the model is fit over the training data, our model is ready for use.
Although Support Vector Regression is used rarely it carries certain advantages that are as mentioned below:
It is robust to outliers.Decision model can be easily updated.It has excellent generalization capability, with high prediction accuracy.Its implementation is easy.
It is robust to outliers.
Decision model can be easily updated.
It has excellent generalization capability, with high prediction accuracy.
Its implementation is easy.
Some of the drawbacks faced by Support Vector Machines while handling regression problems are as mentioned below:
They are not suitable for large datasets.In cases where the number of features for each data point exceeds the number of training data samples, the SVM will underperform.The Decision model does not perform very well when the data set has more noise i.e. target classes are overlapping..
They are not suitable for large datasets.
In cases where the number of features for each data point exceeds the number of training data samples, the SVM will underperform.
The Decision model does not perform very well when the data set has more noise i.e. target classes are overlapping..
With that, we have reached the end of this article. I hope this article would have helped you get a feel about the idea behind SVR algorithms. If you have any question or if you believe I have made any mistake, please contact me! You can get in touch with me via: Email or LinkedIn.