Support Vector Machine (SVM)
By: Arneesh Aima
Tags: Support Vector Machine, Artificial Intelligence, Machine Learning, Data Science, Svm

[1] Introduction and Methodology

SVM stands for Support Vector machine, it is an algorithm which is used for classification of linear and non-linear data. A separation between two classes is established using plane called “Hyperplane”. The main aim is to obtain the optimum hyperplane for data point separation into classes. If the data points can be classified using a simple linear hyper plane it is the case of linear SVM. However if the data points can’t be classified into classes using a linear classifier then a transformation function is first applied to the dataset to relocate the data points of one class to other coordinates or higher dimension and then after the relocation process the data points are separated using the same old linear classifier. The hyperplane for the separation of data points is obtained using the help of support vectors and margins. Support vectors are some selected data points with lie on the edge of either classes. Through these support vectors margins are established along the support vectors. The separation space between the data points is called “Gutter Space”.

[2] Key Points :

Support Vectors : it is an input vector that just touches the boundary of the margin

it is an input vector that just touches the boundary of the margin Margins : the imaginary boundary created with the help of support vectors.

the imaginary boundary created with the help of support vectors. There are two distances in SVM : d+ and d- . d+ is on the R.H.S of the separating hyperplane and d- is on the L.H.S of the separating hyperplane.

Our main aim in SVM is to “maximize the margins”.

[3] Process For Linearly separable data points :

Data points for class A say positive class and data points for class B say negative class are taken

The data points are plotted on a graph as per the coordinates of axes , say x1 and x2

By inspection we figure out how many Support Vectors are required for separation of data

All the vectors are augmented with a bias of 1. For example s1 (1 2) becomes s˜1 (1 2 1)

We need to find out values of αi for each of the vectors

In this case of Linear SVM the simple mapping function Φ() is used to obtains equations:

α1Φ(s1) · Φ(s1) + α2Φ(s2) · Φ(s1) + α3Φ(s3) · Φ(s1) = v1 α1Φ(s1) · Φ(s2) + α2Φ(s2) · Φ(s2) + α3Φ(s3) · Φ(s2) = v2 α1Φ(s1) · Φ(s3) + α2Φ(s2) · Φ(s3) + α3Φ(s3) · Φ(s3) = v3

Here the values for v1 , v2 and v3 is obtained as per the support vectors and margins chosen. A threshold is established in the gutter space and all the values right of the region and given +ve values and all the values on the left hand side of the threshold are given –ve values. Similar concept is followed in the case if the margins obtained are horizon, in that case all the values above the thresh hold are considered +ve and all the values below the threshold are granted –ve values.

The mapping function in each equation is each replaced by its corresponding support vector

α1s˜1 · s˜1 + α2s˜2 · s˜1 + α3s˜3 · s˜1 = v1

α1s˜1 · s˜2 + α2s˜2 · s˜2 + α3s˜3 · s˜2 = v2

α1s˜1 · s˜3 + α2s˜2 · s˜3 + α3s˜3 · s˜3 = v3

Values of augmented support vectors are placed in these equations along with the values of v1,v2 and v3 as per the condition and then the equations are solved for obtaining the values of α1, α2 and α3.

Obtain the value coordinates of the final hyper plane using

w˜ =∑ i αis˜i

The first two values in the final obtained vector denote the hyper plane x1 and x2 coordinates and the third value of the vector is the offset.

The separating hyper plane equation is:

y = wx+b

Where : w = (x1 x2 ) and b = offset

[4] Process For Non — Linearly separable data points

Data points for class A say positive class and data points for class B say negative class are taken

The data points are plotted on a graph as per the coordinates of axes , say x1 and x2

By inspection we notice that this data cant be separated using a straight line , so a transformation mapping function needs to be applied to this dataset in order to shift coordinates of one call to higher dimension

Transformation Function

Where :

x1 : coordinate of 1st axis

x2: coordinate of 2nd axis

v : some value obtained from inspection of data set . For example if the support vectors have values like s1( 1 3) , s2(1, 2) and so on . , in that case the value of v can be set a v=6.

New modified support vectors are obtained using this transformation function. Now these transformed data points can be separated using Linear SVM

All the transformed vectors are augmented with a bias of 1. For example s1 (1 2) becomes

s˜1 (1 2 1)

We need to find out values of αi for each of the vectors

Φ() is used to obtains equations:

α1Φ(s1) · Φ(s1) + α2Φ(s2) · Φ(s1) + α3Φ(s3) · Φ(s1) = v1 α1Φ(s1) · Φ(s2) + α2Φ(s2) · Φ(s2) + α3Φ(s3) · Φ(s2) = v2 α1Φ(s1) · Φ(s3) + α2Φ(s2) · Φ(s3) + α3Φ(s3) · Φ(s3) = v3

Here the values for v1 , v2 and v3 is obtained as per the support vectors and margins chosen. A threshold is established in the gutter space and all the values right of the region and given +ve values and all the values on the left hand side of the threshold are given –ve values. Similar concept is followed in the case if the margins obtained are horizon, in that case all the values above the thresh hold are considered +ve and all the values below the threshold are granted –ve values.

The mapping function in each equation is each replaced by its corresponding support vector

α1s˜1 · s˜1 + α2s˜2 · s˜1 + α3s˜3 · s˜1 = v1 α1s˜1 · s˜2 + α2s˜2 · s˜2 + α3s˜3 · s˜2 = v2 α1s˜1 · s˜3 + α2s˜2 · s˜3 + α3s˜3 · s˜3 = v3

Values of augmented support vectors are placed in these equations along with the values of v1,v2 and v3 as per the condition and then the equations are solved for obtaining the values of α1, α2 and α3.

Obtain the value coordinates of the final hyper plane using

w˜ =∑ i αis˜i

The first two values in the final obtained vector denote the hyper plane x1 and x2 coordinates and the third value of the vector is the offset.

The separating hyper plane equation is:

y = wx+b

Where : w = (x1 x2 ) and b = offset

[5] Kernel Trick

Till now the data points we used could be either separated using a linear hyperplane or a transformation function was used to transform then to high dimension ( in previous cases to higher coordinates ). However in various cases these methods are not sufficient for separation of positive and negative data points, in those cases we need to use a transformation function which transforms the data points to comparatively higher dimension than before . i.e to another plane altogether. This transformation function transforms our data to 3-dimensional feature space.

Kernel Trick for SVM

Where the values for v1 and v2 are chose by inspection of the coordinates of data points. Values of v1 and v2 should be in accordance with the coordinates which we need to segregate.

When we are using the kernel trick in SVM the augmentation of vectors with bias 1 is not essential, however we can augment it if we desire but it would only increase the complexity of the equations to be solved.

[6] Numerical on Support Vector Machine : Linearly Separable

Positively labeled points : (4 1) , (4 -1) , (5 1) , (5 -1)

Negatively labeled points : (2 0) , (0 2), (0 -2) , (-2 0)

Plotting the points on a graph taking axes -> x1 and x2

Selecting support vectors by inspection:

s1=(2 0) s2=(4 1) s3=(4 -1)

Augmentation of support vectors :

s~1=(2 0 1) s~2=(4 1 1) s~3=(4 -1 1)

Formation of equations:

α1s˜1 · s˜1 + α2s˜2 · s˜1 + α3s˜3 · s˜1 = -1 α1s˜1 · s˜2 + α2s˜2 · s˜2 + α3s˜3 · s˜2 = 1 α1s˜1 · s˜3 + α2s˜2 · s˜3 + α3s˜3 · s˜3 = 1

Substituting values of s~1 , s~2 and s~3 in the above equations :

α1(2 0 1) (2 0 1) + α2(4 1 1) (2 0 1) + α3(4 -1 1)(2 0 1) = -1

α1(4+0+1) + α2(8+0+1) + α3(8–0+1)=-1

5α1+ 9α2 + 9α3=-1

α1(2 0 1)(4 1 1) + α2(4 1 1)(4 1 1) + α3(4 -1 1)(4 1 1) = 1

α1(8+0+1) + α2(16+1+1) + α3(16–1+1) = 1

9α1 + 18α2 + 16α3 = 1

α1(2 0 1)(4 -1 1) + α2(4 1 1)(4 -1 1) + α3(4 -1 1)(4 -1 1) = 1

α1(8+0+1)+ α2(16–1+1) + α3(16+1+1) = 1

9α1 + 16α2 + 18α3= 1

Solving the above equations we get values :

α1= -6.5 α2= 1.75 α3= 1.75

Using — w˜ =∑ i αis˜i

= -6.5(2 0 1) + 1.75(4 1 1) + 1.75(4 -1 1)

= -13+7+7= 1

=0+1.75–1.75= 0

= -6.5+1.75+1.75= -3

We Obtain ( 1 0 3)

Comparing with : y = wx +b

We have -> w= (1 0) : Hyperplane

B= -3 : Bias

[7] Numerical on Support Vector Machine : Non Linearly Separable ( Transformation function required )

Positively labeled points : (3 0) , (0 -3) , (-3 0) , (0 3)

Negatively labeled points : (1 1) , (1 -1), (-1 -1) , (-1 1)

Plotting the points on a graph taking axes -> x1 and x2

Selecting support vectors by inspection:

s1=(1 1) s2=(15 12) s3=(12 15)

Augmentation of support vectors :

s~1=(1 1 1) s~2=(15 12 1) s~3=(12 15 1)

Formation of equations:

α1s˜1 · s˜1 + α2s˜2 · s˜1 + α3s˜3 · s˜1 = -1 α1s˜1 · s˜2 + α2s˜2 · s˜2 + α3s˜3 · s˜2 = 1 α1s˜1 · s˜3 + α2s˜2 · s˜3 + α3s˜3 · s˜3 = 1

Substituting values of s~1 , s~2 and s~3 in the above equations :

α1(1 1 1) (1 1 1) + α2(15 12 1) (1 1 1) + α3(12 15 1)(1 1 1) = -1

α1(1+1+1) + α2(15+12+1) + α3(12+15+1)=-1

3α1+ 28α2 + 28α3=-1

α1(1 1 1)(15 12 1) + α2(15 12 1)(15 12 1) + α3(12 15 1)(15 12 1) = 1

α1(15+12+1) + α2(225+144+1) + α3(180+180+1) = 1

28α1 + 370α2 + 361α3 = 1

α1(1 1 1)(12 15 1) + α2(15 12 1)(12 15 1) + α3(12 15 1)(12 15 1) = 1

α1(12+15+1)+ α2(180+180+1) + α3(144+225+1) = 1

28α1 + 361α2 + 371α3= 1

Solving the above equations we get values :

α1= -1.259 α2= 0.049 α3= 0.049

Using — w˜ =∑ i αis˜i

= -1.259(1 1 1) + 0.049(15 12 1) + 0.049(12 15 1)

= -1.259+0.735+0.588= 0.064

=-1.259+0.588+0.735= 0.064

= -1.259+0.049+0.049= -1.161

We Obtain ( 0.064 0.064 -1.161)

Normalize the above vector by dividing by : 0.064

Now we get : ( 1 1 -18.140)

Comparing with : y = wx +b We have -> w= (1 1) : Hyperplane B= -18.140 : Bias



My LinkedIn : Thank You !My LinkedIn : Visit Me on LinkedIn

Gain Access to Expert View — Subscribe to DDI Intel