Linear Regression from Scratch with Tensorflow 2 | by Cedric Conol | Towards Data Science

Linear regression is one of the most basic and perhaps one of most commonly used machine learning algorithm that beginners and experts alike should know by heart. In this article I will walk you through how to implement linear regression using only Tensorflow. The discussion will be divided into two parts, the first part explains the concept of linear regression, the second part is a walk through of how to implement linear regression in Tensorflow.
Linear regression attempts to model the relation of dependent and independent variables by fitting a linear equation. Suppose we have the data of quiz scores and the length of study hours of 100 students.
By visually inspecting the scatter plot, we can easily draw a line with the equation y=mx+b , where m and b are the slope and y-intercept, respectively. These can be seen on the figure as roughly 40 and 0 for m and b respectively.
Let’s draw a line where m=40 and b=0.
The y=40x line looks good! We can then estimate that a student’s score is just 40 multiplied by the number of hours a student studied.
Linear regression works exactly like this except that it cannot visually check the slope and y-intercept from the scatter plot. What it does instead is to guess the slope and y-intercept first then measure how good its guess is. If it’s not good enough then it adjusts the slope and y-intercept until the line fits the data well.
Linear regression is a three-step algorithm:
Initialize the parameters of a linear equation (first guess of slope and y-intercept).Measure the goodness of fit based on some function.Adjust the parameters until the measure in step 2 looks good.
Initialize the parameters of a linear equation (first guess of slope and y-intercept).
Measure the goodness of fit based on some function.
Adjust the parameters until the measure in step 2 looks good.
Now that we built our intuition around linear regression, let’s talk about each of the step mathematically.
The first step in linear regression model is to initialize a linear equation, yes, we’ll use y=mx+b but we have to generalize our approach. The reason for such is that we might be facing data with multiple independent variables. Think of it as adding another variable in our Quiz Score data, like the amount of coffee consumed while studying. Having this coffee dimension will make the linear equation look like this: y=m1x1+m2x2+b , where m1 and m2 are slopes for the study hours and coffee dimensions, respectively, and x1 and x2 are the study hours and coffee variables.
We’ll use dot product to represent the product of matrices, m and x, instead of writing a longer equation for every new variable. Note that it is also valid to use the term, tensor, since it is the generalization of a matrix. Bold type letters are used to denote matrices, so the linear equation should be written as y=m⋅x+b.
There are many ways to initialize the parameters of the equation, the most common ones are using random values, zeros, or ones. You are free to use any type of initialization, this choice will determine how fast your learning algorithm terminates. In the next iteration of the algorithm, these parameters will be updated based on some function discussed in step 2.
Now let’s say the initial values you set for m and b are ones, so your equation is y=1x+1 . The initial predictions will look like the orange dots in the figure below. It’s obviously a very bad prediction, we need a number to quantify how bad or good these predictions are.
There are many ways to measure the goodness of our prediction, we’ll use one of them called mean squared error (MSE). In this context, error means difference, so MSE literally means taking the square of the difference between actual and predicted values, then take the average. It is written mathematically as
Functions like MSE are called loss function or objective function. These are functions that the algorithm wants to minimize. If our linear regression model perfectly predicts the quiz scores, its MSE will be equal to 0. So in every iteration of the algorithm, it should update the parameters such that the MSE comes closer to 0 without overfitting. Overfitting is an entire topic itself, but what it means essentially is that we don’t want our learning algorithm to be so good with the training data and fail miserably on the test set.
3. Gradient Descent
Sure we can keep on guessing the parameters until we get close enough to zero MSE but it will take time and effort — gradient descent will do this for us. If you’re not familiar with the term, there’s tons of articles and videos explaining its concept.
Gradient descent is one of the building blocks of artificial intelligence. It is the learning in machine learning. Algorithms like gradient descent allows learning algorithms to learn without being told so explicitly. (You need to brush up on your calculus to understand how gradient descent works.)
Gradient descent is an optimization algorithm that we will use to minimize our loss function (MSE in this case). It does so by updating the parameters with small changes in each iteration, this change can be big too depending on your preference (learning rate).
In each new iteration, the updated parameters will be p_new=p_old-(l*dL/dp), where p is the parameter, which could be the slope, m, or the y-intercept, b. The new variables, l and dL/dp, are the learning rate and partial derivative of the loss function with respect to the parameter.
With enough iterations, the slope and y-intercept will get closer to 40 and 0, the values we consider to be “close enough” to fit to our data. As you may have observed, if you happen to initialize the parameters close to 40 and 0, say 35 and 0.5, then the algorithm will take less iterations.
This article is very helpful if you want to dig deeper into the math of gradient descent.
Here are some of the possible ways to terminate the algorithm:
Terminate the algorithm once a specified number of iterations is satisfied.Terminate the algorithm once a specified MSE is satisfied.Terminate the algorithm if the MSE does not improve in the next iteration. You can specify a precision such as 0.001, if the difference between two successive MSEs is less than this precision, then stop the algorithm.
Terminate the algorithm once a specified number of iterations is satisfied.
Terminate the algorithm once a specified MSE is satisfied.
Terminate the algorithm if the MSE does not improve in the next iteration. You can specify a precision such as 0.001, if the difference between two successive MSEs is less than this precision, then stop the algorithm.
In this demo, we’ll follow the three-step approach to linear regression algorithm discussed above and terminate the algorithm using stopping criteria 1.
These are the only libraries we’ll need for this demo. TensorFlow for building the algorithm, pyplot for visualization purposes, and boston_housing as our toy dataset.
import tensorflow as tfimport matplotlib.pyplot as pltfrom tensorflow.keras.datasets import boston_housing
Let’s start off by creating aSimpleLinearRegression class with initialization options.
I specified three initialization options, ones , zeros , and random (default). tf.random.uniform will generate a tensor with random values from uniform distribution within the range minval and maxval with shape shape . I defined m as a variable with no particular shape so it can be flexible enough to accept any number of independent variables, this can be done by setting shape=tf.TensorShape(None) .
Next is to implement our loss function, MSE. To recap, MSE is written mathematically as:
Here’s my implementation of the function in TensorFlow:
This is fairly easy to write in Tensorflow. First you take the difference of true and predicted value, use tf.square to square the differences then get the mean of the squared differences using tf.reduce_mean .
This function accepts true and predicted values, the former comes from the data itself, but the latter have to be computed.
The predict method is done by simplifying the linear equation. First we take the dot product of m (slope tensor) and x (feature tensor) and add the y-intercept b . I had to specify the axis to which the reduction in reduction_sum will be computed to 1 , otherwise it will reduce the tensor to a single sum.
We need gradient descent to update our parameters for every iteration. There’s no need to create this algorithm from scratch as Tensorflow has a built it function for this, tf.GradientTape . By default, GradientTape has persistent set to False which means at most one call can be made to the gradient() method in this object. Since we are using this to calculate for two gradients (one for m and another for b ) for every iteration, we have to set this to True. Then we specify the loss function to which gradients will be computed, in this case mse with parameters y and self.predict(X) which represents true and predicted values respectively.
Each parameter will be updated by subtracting the product of learning rate and the gradient of the parameter.
p_new = p_old - (l*dL/dp)
Learning is a hyperparameter and should be specified when training the linear regressor. The gradient, dL/dp , is computed using the gradient() method, which accepts the loss function and the parameter. This operation just solves for the partial derivative of the loss function with respect to the parameter. We have to compute for two gradients in each iteration, one for m and b .
To update the parameter with the new value, which is just the old value minus l*dL/dp , we simply use assign_sub() method of tf.Variable .
Let’s put everything together in a train method.
First thing that it does is to check if the data only contain a single independent variable, if it is, then it will reshape it to become a 2D tensor.
self.m.assign([self.var]*X.shape[-1]) will initialize m with initial values we set during initialization with a shape following the number of independent variables the data have.
The stopping criteria of our algorithm is the number of iteration, defined by epoch . For each iteration, it will call the update method.
It’s time to test our algorithm using the Boston Housing dataset.
Load the dataset using keras.datasets :
(x_train, y_train), (x_test, y_test) = boston_housing.load_data()
Standardize the data:
mean_label = y_train.mean(axis=0)std_label = y_train.std(axis=0)mean_feat = x_train.mean(axis=0)std_feat = x_train.std(axis=0)x_train = (x_train-mean_feat)/std_featy_train = (y_train-mean_label)/std_label
Create and train a SimpleLinearRegression object.
linear_model = SimpleLinearRegression('zeros')linear_model.train(x_train_norm, y_train_norm, learning_rate=0.1, epochs=50)
Here’s the losses of the last five iteration:
Let’s predict using the test set:
# standardizex_test = (x_test-mean_feat)/std_feat# reverse standardizationpred = linear_model.predict(x_test)pred *= std_labelpred += mean_label
We have to standardize the input first then reverse the process once we have the prediction.
Despite its simplicity, linear regression is one of the most commonly used machine learning algorithms in the industry, and some companies test how well you understand it. Although there are much simpler ways to implement this algorithm, like using scikit-learn and even TensorFlow’s LinearRegressor , we implemented the entire algorithm from scratch with the intent of experiencing TensorFlow’s functions. When you go into neural networks, there are high-level libraries like Keras which simplifies the process of building neural networks using TensorFlow as backend. But eventually, especially if you are a researcher, you would want to customize your models however you want it, and this is where the low-level functions of TensorFlow are very useful.