Chapter 1.3:Code for linear regression with Gradient descent (from scratch and Tensorflow & Scikit Learn ).
By: Madhu Sanjeevi, Mady
Tags: Supervised Learning, Machine Learning, TensorFlow, Scikit Learn, Gradient Descent

In last two parts we talked about Linear regression and Gradient descent.

This part I wanna code the algorithm from scratch and using other two libraries (Tensorflow and scikit learn). Lets get started!

Full code is available on my Github.

First things first, we should prepare the data-set , we can generate random data X and y using Numpy and plot it.

Sine wave random

Now we have the X values and y values (inputs and targets), next step is to generate random weights (ϴ values)

We got our ϴ values(ϴ0 and ϴ1) and X values , next step is to predict y values(remember these are different from actual y values from our data-set)

y=ϴ0+ϴ1*X

Dot operation for matrix multiplication.

the above code does matrix multiplication same as y1=x0*ϴ0+x1*ϴ1 (x0=1, x1=our first X value in data-set). next step is to calculate the error and minimize the error (Gradient descent process)

error=(predicted-actual)**2 and , to minimize the error we need to find the derivatives(gradients) for weights(ϴ values) and update them as we reach the minimum value .

After iterating it by no of epochs, we finally get the final weights (best fit line for our data) here is the graph for error with epochs

As we can see the error is reduced as no of epochs are increasing and final we reach the minimum value, and we got our updated weights and best fit line.

Best fit line

What’s next??

Next step is to predict the new data we can simply do that by just calling the predict() function (Weights W are final weights) by giving the set of test data (X values)

we can also add many features to X (no of features can be more than one), create a data set with multiple features in X and run the same program

it will work.

Lets use some other frameworks (tensorflow and scikit learn)

#Tensorflow

import tensorflow as tf x = tf.placeholder(tf.float32)

y = tf.placeholder(tf.float32) tf.set_random_seed(5) w = tf.Variable(tf.random_normal([1]),name="weights")

b = tf.Variable(tf.random_normal([1]), name='bias')

placeholders for the input and output of the network. Placeholders are variables which we need to fill in when we are ready to compute the graph.

Next we predict the y values

Y_Pred = tf.add(tf.multiply(X, W), b) # same as ϴ1*X + ϴ0

Now calculate the error for every X value and sum it up for the final error

error=tf.square(Y_Pred-Y)

f_error=tf.reduce_sum(error)/(points-1) #we just reduced error by dividing the no of X values (Good practice)

Now update the weights to reduce the error , how? Gradient descent

Tensorflow has already implemented GD so we don’t need to write it.

optimizer=tf.train.GradientDescentOptimizer(0.01).minimize(f_error)

Note: we need to create a session in tensorflow to run the code

Now We can do the same thing using scikit learn in just a few line of code , how awesome is that?

Oh! yeah! finally we made the same algorithm work in 3 different ways

here is the final picture comparing the best fit line for all 3 methods

From scratch(Left),Tensorflow(Center),scikit(Right)

Can you guess which one is best??? let me know what you think

That’s it for this story. Hope you changed and learned something let me know if it helps .

In the next story I will cover another interesting machine leaning concept until then See Ya!

Full code is available on my Github