Simple Linear Regression with Scratch
By: Akif Emre Çapoğlu
Tags: Machine Learning, Scratch, Deep Learning, Linear Regression

Hello. Welcome to Simple Linear Regression with Scratch, the first article in our Machine Learning with Scratch series. If we need to examine the data we will use

These are our data.

Simple Linear Regression

The formula at the focus of our code is the formula above. To explain the formula, if the value we want to estimate “y” is B0 “Bias”, it is the fixed coefficient value called its value. B1 is the coefficient value to accompany our inputs. While writing the algorithm, our aim is to find the best B0 and B1 values we can find.

“To-do list for those who want to write without looking at the code”

Upload your data Divide your data into two parts as train and test We discard random values of B0 and B1. (I found the value 0 suitable for this code.) We run our train function. We find the optimum value of B0 and B1 with the Gradient Descent algorithm in the Train function. We estimate the new values using the B0 and B1 values we found.

What is Our Function?

The loss function is a unit of measure that measures the proximity or distance of values predicted by our model to actual values. In this code, we used the Mean Squared Error (MSE) function to minimize it.

Mean Squared Error

Here ȳᵢ is the value we estimate and yi is the values we hold to estimate in our train data.

What is Gradient Descent?

Gradient Descent is an optimization algorithm that will minimize our lost function. In order to understand Gradient Descent better, I strongly recommend you to listen to the gradient descent series explained by our Dr. Uzak Çetin.

Things to do for the gradient descent algorithm

Determining starting weight Take the derivative of the cost function for each parameter Update Weight and Bias values

While updating the rules, I think that the “a” value there draws your attention. So what is the “a” value? This coefficient, called the Learning Rate, is the coefficient that measures the sensitivity level of the job while updating the rules. So how should we choose the coefficient “a”?

As you can see above, the learning rate coefficient has an important role in our function. If we choose too high, it cannot find the minimum value of the loss function as in the “Too High” section. If we choose too low, our function will have to repeat a large number of times. Commonly used values are [0.01, 0.001, 0.02, 0.03, 0.005] but using various values may be appropriate for your function.

Let’s move on to our code.

Let’s import our libraries first

2. Upload Your Data

To explain our code; While our x variable will be the input part of the data we will train, our y variable is the output values to be used in the loss function while training. While our X1 value is the input part of our predicted data, X2 is the data we have to calculate the MSE score of our model. After defining the data, we can assign the “class” structure we will give below to the “model” variable and perform our operations from our “model” variable.

3. Let’s code our class structure

Our “Class” structure consists of 3 different parts. The first is our __init__ function, which we will predefine. We start by defining the variables we will generally use in the __init__ part. In the second part, we take the loop as many times as we want it to rotate. In our next cycle, we decide how much to turn for our gradient descent application. In this loop, we first derive the loss function for each variable. When we multiply our learning rate coefficient with our derivative variables and subtract it from the current weight and bias values, we update our weight and bias values. As the next step, we will try to predict new inputs by using the best weight and bias values that we determined with gradient descent in the third step.

For code : https://github.com/capogluuu/ml_scratch