Support Vector Regression (SVR) Model: A Regression-Based Machine Learning Approach
By: Abhilash Singh
Tags: Wireless Sensor Network, Support Vector Regression, Regression, Machine Learning

In this article, we will briefly discuss the SVR model. We will discuss three types of SVR namely, S-SVR (Scaling-SVR), Z-SVR (Z-score-SVR), and R-SVR (Range-SVR). Afterwards, we will discuss its application in predicting the Average Localisation Error (ALE) in node localisation process in Wireless Sensor Network (WSNs).

You can download our paper for more details. You can write to me (abhilash.singh@ieee.org)if you have any question or visit my web page for more updates.

Introduction

Broadly, learning algorithms are divided into supervised and unsupervised learning. Further, supervised learning is classified into classification and regression learning, whereas unsupervised learning is classified into clustering and dimension reduction techniques.

The key objective of regression-based machine learning algorithms is to predict the predictand based on a mapping function. This mapping function is modelled by feeding a set of features and predictand data known as training data set. SVR is used in many applications such as image processing, remote sensing, and blockchain. It has superb generalisation competence along with high accuracy. Also, the computational complexity is independent of the input feature data set.

Let us see the governing equations of SVR in the next section;

Support Vector Regression

SVR was initially proposed by Drucker et al., which is a supervised learning technique, based on the concept of Vapnik’s support vectors. SVR aims at reducing the error by determining the hyperplane and minimising the range between the predicted and the observed values. Minimising the value of w in the Equation given below is similar to the value defined to maximise the margin, as shown in Fig. 1.

where the summation part represents an empirical error. Hence, to minimise this error, we use the following equation.

where the alpha term represents the Lagrange multiplier and its value is greater than equal to 1. K represents the kernel function and B represents the bias term. In this study, we have used the Polynomial kernel given by:

Where d is the polynomial degree and γ is the polynomial constant.

SVR performs better performance prediction than other algorithms like Linear Regression, KNN and Elastic Net, due to the improved optimisation strategies for a broad set of variables. Moreover, it is also flexible in dealing with geometry, transmission, data generalisation and additional functionality of kernel. This additional functionality enhances the model capacity for predictions by considering the quality of features.

The training samples influence the SVR model’s fitting performance since the SVR algorithm is sensitive to the interference in the training data. Besides, SVR is useful in resolving high dimensional features regression problem, and well-function if the feature metrics is larger than the size of the sample. In this study, we have extracted four features, namely anchor ratio, transmission range, node density and the number of iterations from modified CS algorithm simulation.

Feature scaling is essential for SVR because, when one function has greater magnitudes than others, the other features will dominate while measuring the distance. To avoid this, we have used various standardisation approaches. Based on this, we have proposed three methods, as shown in Fig. 2.

The method I is S-SVR (Scaling SVR). In this method, we first standardised the features using the Equation given below;

Where x is the feature vector, xs is the standardised data, and σ is the standard deviation of the feature vector. The method II is Z-SVR (Z-score SVR). In this method, we have standardised the features using the Equation given below;

Where x bar is the mean of the feature vector. The method III is the R-SVR (Range SVR). In this method, we have standardised the features using the Equation given below;

Afterwards, we trained and tested the SVR models in a 70:30 ratio, as shown in Fig. 2. In this study, the dimension of the features vector is 107 × 1. Hence, we have used 75 data for training and the remaining 32 for testing.

Application of SVR for predicting ALE in node localisation in WSNs.

For simulating the SVR model, we performed the hyperparameter tuning through the grid search algorithm. In doing so, we fixed one of the hyper-parameter (i.e., epsilon at 0.01) and applied the grid search algorithm to find the value of the other hyper-parameter (i.e., C). We created a 100 × 100 grid for the penalty factor, C. Each grid represents a specific value of C. On simulating the grid search algorithm, it finds an optimal grid that corresponds to the minimum value of the MSE. The range of optimal C for all three methods along with the other simulation parameter value is given in the table below.

Table 1: Simulation parameters for SVR

Results

In this section, we have presented the results of the method I, II and III for ALE prediction in the respective subsections. We have plotted a linear regression curve between the predicted ALE and the simulated ALE for comparison.

Performance of method I

Fig. 3: Prediction results for ALE using method I.

We have compared the predicted ALE results, thus obtained by method I with the simulated results of the modified CS algorithm. We found that predicted results accorded well with the simulated results and gathered along the straight regression line with mild scattering (Fig. 3). The shaded grey region corresponds to the 95% Confidence Interval (CI) of the regression line and suggests that the predicted result has a strong positive correlation with R = 0.80 and RMSE = 0.23m.

Fig. 4: Prediction results for ALE using method II.

Once we calculated the predicted ALE through method II, we have evaluated its performance with the simulated results of the modified CS algorithm. In doing so, we found a good agreement between both with R = 0.81 and RMSE = 0.20m (Fig. 4). However, some observed values lie outside the CI of the regression line due to the overestimation of the ALE value by the SVR model. The overestimation probably occurs due to the positive bias. This type of error comes under systematic error which is mainly due to the model or approach used.

Fig. 5: Prediction results for ALE using method II.

We have compared the predicted ALE of method III with the simulated ALE obtained through a modified CS algorithm. In this case, also, we found a strong correlation between the variables (Fig. 5). Here, we found a pragmatic correlation of R = 0.82 with RMSE = 0.15m.

Conclusion

In this article, we discussed and investigated three SVR based machine learning model for ALE prediction. These methods are defined based on the standardisation method used. In the method I, II and III, we have used scaling, Z-score and range standardisation methods respectively. Afterwards, we trained the SVR model with the polynomial kernel using the standardised data and evaluated its performance using correlation of coefficient and RMSE metrics. In doing so, we found that range standardisation of the features (i.e., method III) results in lower RMSE in ALE prediction. Also, the coefficient of correlation is highest in method III.

References