TensorFlow: Reduction to Linear Regression
By: Bhanu Soni
Tags: Support Vector Machine, Machine Learning, Deep Learning, Linear Regression, TensorFlow

TensorFlow: Reduction to Linear Regression

Support vector machines can be used to fit the linear regression

We will explore how Support vector machines can be used to fit the linear regression with TensorFlow.

Introduction

Support vector machines are a method of binary classification. The basic idea is to find a linear separating line (or hyperplane) between the two classes. Linear Regression is a linear approximation of a causal relationship between 2 or more variables.

Getting ready

Following our previous article, TensorFlow: Support Vector Machines, The same maximum margin concept can be applied toward fitting linear regression. Instead of maximizing the margin that separates the classes, we can think about maximizing the margin that contains the most (x, y) points. To illustrate this, we will use the same iris data set, and show that we can use this concept to fit a line between sepal length and petal width.

The corresponding loss function will be similar to max

Here, ε is half of the width of the margin, which makes the loss equal to zero if a point lies in this region.

How to do it…

First, we load the necessary libraries, start a graph, and load the iris dataset. After that, we will split the dataset into train and test sets to visualize the loss on both. Use the following code:

import matplotlib.pyplot as plt

import numpy as np

import tensorflow as tf

from sklearn import datasets sess = tf.Session()

iris = datasets.load_iris() x_vals = np.array([x[3] for x in iris.data])

y_vals = np.array([y[0] for y in iris.data]) train_indices = np.random.choice(len(x_vals), round(len(x_

vals)*0.8), replace=False)

test_indices = np.array(list(set(range(len(x_vals))) - set(train_

indices))) x_vals_train = x_vals[train_indices]

x_vals_test = x_vals[test_indices]

y_vals_train = y_vals[train_indices]

y_vals_test = y_vals[test_indices]

For now, we have split the data into train and test. It is also common to split the data into three datasets, which includes the validation set. We can use this validation set to verify that we are not overfitting models as we train them.

2. Let’s declare our batch size, placeholders, and variables, and create our linear model, as follows:

batch_size = 50 x_data = tf.placeholder(shape=[None, 1], dtype=tf.float32)

y_target = tf.placeholder(shape=[None, 1], dtype=tf.float32) A = tf.Variable(tf.random_normal(shape=[1,1]))

b = tf.Variable(tf.random_normal(shape=[1,1])) model_output = tf.add(tf.matmul(x_data, A), b)

3. Now we declare our loss function. The loss function, as described in the preceding text, is implemented to follow with ε=0.5:. Remember that the epsilon is part of our loss function, which allows for a soft margin instead of a hard margin.

epsilon = tf.constant([0.5])

loss = tf.reduce_mean(tf.maximum(0., tf.sub(tf.abs(tf.sub(model_

output, y_target)), epsilon)))

4. We create an optimizer and initialize our variables next, as follows:

my_opt = tf.train.GradientDescentOptimizer(0.075)

train_step = my_opt.minimize(loss) init = tf.initialize_all_variables()

sess.run(init)

5. Now we iterate through 200 training iterations and save the training and test loss for plotting later:

train_loss = []

test_loss = [] for i in range(200):

rand_index = np.random.choice(len(x_vals_train),size=batch_size)

rand_x = np.transpose([x_vals_train[rand_index]])

rand_y = np.transpose([y_vals_train[rand_index]])

sess.run(train_step, feed_dict={x_data: rand_x,y_target:rand_y})



temp_train_loss = sess.run(loss, feed_dict={x_data: np.transpose([x_vals_train]), y_target: np.transpose([y_vals_

train])})

train_loss.append(temp_train_loss) temp_test_loss = sess.run(loss, feed_dict={x_data:

np.transpose([x_vals_test]), y_target: np.transpose([y_vals_

test])})

test_loss.append(temp_test_loss)

if (i+1)%50==0:

print('-----------')

print('Generation: ' + str(i))

print('A = ' + str(sess.run(A)) + ' b = ' +str(sess.run(b)))

print('Train Loss = ' + str(temp_train_loss))

print('Test Loss = ' + str(temp_test_loss))

6. This results in the following output:

Generation: 50

A = [[ 2.20651722]] b = [[ 2.71290684]]

Train Loss = 0.609453

Test Loss = 0.460152

-----------

Generation: 100

A = [[ 1.6440177]] b = [[ 3.75240564]]

Train Loss = 0.242519

Test Loss = 0.208901

-----------

Generation: 150

A = [[ 1.27711761]] b = [[ 4.3149066]]

Train Loss = 0.108192

Test Loss = 0.119284

-----------

Generation: 200

A = [[ 1.05271816]] b = [[ 4.53690529]]

Train Loss = 0.0799957

Test Loss = 0.107551

7. We can now extract the coefficients we found, and get values for the best-fit line. For plotting purposes, we will also get values for the margins as well. Use the following code:

[[slope]] = sess.run(A)

[[y_intercept]] = sess.run(b)

[width] = sess.run(epsilon) best_fit = []

best_fit_upper = []

best_fit_lower = []

for i in x_vals:

best_fit.append(slope*i+y_intercept)

best_fit_upper.append(slope*i+y_intercept+width)

best_fit_lower.append(slope*i+y_intercept-width)

8. Finally, here is the code to plot the data with the fitted line and the train-test loss:

plt.plot(x_vals, y_vals, 'o', label='Data Points')

plt.plot(x_vals, best_fit, 'r-', label='SVM Regression Line',

linewidth=3)

plt.plot(x_vals, best_fit_upper, 'r--', linewidth=2)

plt.plot(x_vals, best_fit_lower, 'r--', linewidth=2)

plt.ylim([0, 10])

plt.legend(loc='lower right')

plt.title('Sepal Length vs Pedal Width')

plt.xlabel('Pedal Width')

plt.ylabel('Sepal Length')

plt.show()

SVM regression with a 0.5 margin on the iris data (sepal length versus petal width)

Here is the train and test loss over the training iterations:

plt.plot(train_loss, 'k-', label='Train Set Loss')

plt.plot(test_loss, 'r--', label='Test Set Loss')

plt.title('L2 Loss per Generation')

plt.xlabel('Generation')

plt.ylabel('L2 Loss')

plt.legend(loc='upper right')

plt.show()

SVM regression loss per generation on both the train and test sets

How it works

Intuitively, we can think of SVM regression as a function that is trying to fit as many points in the 2ε width margin from the line as possible. The fitting of this line is somewhat sensitive to this parameter. If we choose too small an epsilon, the algorithm will not be able to fit many points in the margin. If we choose too large of an epsilon, there will be many lines that are able to fit all the data points in the margin. We prefer a smaller epsilon since nearer points to the margin contribute less loss than further away points.