Gentle Start to Deep Learning implementing Linear Neurons using NumPy
By: Jan Bollenbacher
Tags: Python, Tutorial, Gradient Descent, Deep Learning, Numpy

When starting with Machine Learning and Deep Learning, the number of new vocabularies and concepts can be overwhelming. In this story, we tackle this problem by implementing a linear neuron using just NumPy.

A linear neuron highly relates to the problem of linear regression. The problem we try to solve is equal: We are applying gradient descent on our regression problem to find good weights w.

In this post, we look at the loss function, gradient descent, and training of our model.

Steps to Train a Model

The steps to train a model can be divided into:

Loading the data and preparing our datasets Define the model Training and evaluating the model

For this project and the projects to come, I propose the following directory architecture:

datasets — holding all different datasets we use in our projects

models — containing the various models we use

trainers — having the trainers — the trainer links the models and the datasets together and takes care of the evaluation

main.py — as a starting point for our training and setting the used hyperparameters, this file is being extended as we proceed

Loading the data and preparing our datasets

In this step, we load the data from a file into a format that can be processed (for example, an array). We also need to split our data into different datasets: train and test set. We require both datasets to test that our model doesn’t just fit the training data but generalizes.

Fig. 1 shows two lines: a linear model (blue) and a degree 12 polynomial (red). The polynomial fits the training points perfectly. But if we sample new points, the prediction of the polynomial is very far off. This is called overfitting, and a result of the model being too expressive.

Fig. 1: The black dots are the training samples, and the samples marked with a plus sign are from the test set.

To prevent the model from overfitting, we divide our data into a train and test dataset. With the training dataset, we are training our model to fit the data, and with the test dataset, we check that we don’t overfit the data.

Glimpse into the dataset

In this first attempt, we use a small dataset for regression. This dataset consists of the result of three exams and the final exam score. We want to train our linear neuron to predict the final exam score as close as possible.

A short excerpt of the training data:

Can we predict the final exam score by just knowing the scores of the first three exams?

Code to load the dataset

We add a new file in our datasets folder and name it regression_dataset.py and read the file's data. After that, we split the data into a train and test dataset:

Define the Model

Since we have the data at hand, it is time to think about our linear neuron.

In a linear neuron, every input is multiplied by a weight. The output of the neuron is the sum of all products:

The expressive power of linear neurons is minimal. To learn complex relationships, we need to introduce multiple layers and need neurons that employ some nonlinearity.

A model in TensorFlow 2.x or PyTorch always has the following functions:

Let’s dive into each function individually:

Model Init

Our model consists of one single neuron. We need to generate the weights first. Since we have three inputs, for each exam one, we are also generating three weights:

We hold the values of our weights in a NumPy array and also store the learning rate.

Model Forward

In the forward pass, we calculate our linear neuron's outputs for every row in our dataset. We are doing this by applying the dot product of the input data and our weights.

Matrix X holds our dataset. The first row equals the results of the three exams. We calculate the dot product of X and vector w, holding our weights. If we calculate y1 by hand, we can see that this matches the calculation we want to apply.

Using NumPy, the calculation described above is just one single line of code:

Model Get Loss

We want our linear neuron to output a right prediction of the final exam score. Therefore we need to adapt the weights accordingly. To know if we need to shift the weights, we need to measure our current prediction quality. If the loss is near zero, we don’t need to train our model anymore.

The measurement we are using in the case of regression is Mean Squared Error (MSE).

For each sample i, we calculate the difference between the real outcome y and the predicted outcome y_pred. We square the difference, so higher absolute differences are taken into account more.

To calculate the loss, we first run a forward pass through our model and get a prediction using our current weights. We then use the correct values given in our train dataset and calculate the loss.

Model Fit

We now want to fit our model on our training data. This is done by applying gradient descent. Gradient descent consists of looking at the error the model using the current weights gives us. We are using the derivative of the loss function to find the gradient and then changing the weights gently in the opposite direction. The gradient gives us the slope of the loss function at the point of our current weights. Since the gradient points up the slope, we need to move in the opposite direction to decrease the loss.

We need to calculate the derivative for each dimension of w

We can calculate the gradient using matrices:

The sum of the gradient is higher if the weight has an elevated part of the error.

We then apply the gradient on the weights. To limit the gradient descent step's size, we multiply the gradient with the learning rate beforehand.

Thanks to NumPy, this is just a few lines of code:

Training and evaluating the Model

Training a model consists of a consecutive number of training steps. In each step, we calculate the gradient and apply the data on the model. The weights shift successively in the right direction and will converge to a (local) minimum.

We need to apply several steps on our model weights to find a minimum. Image: https://cdn-images-1.medium.com/max/600/1*iNPHcCxIvcm7RwkRaMTx1g.jpeg

We run a loop over the model fit function and log the loss generated using the test and train dataset. If the train dataset's loss increases, we need to stop training, as we are starting to overfit the data.

Generate a new folder trainers and add the file trainer_linear.py.

The trainer loads the dataset and the model and links everything together.

The starting point for our training

The starting point for our training is file main.py. We set up the trainer here and feed all the necessary hyperparameters. The parameters can also be parameters set by calling the file from the console.

In future articles, we will extend the parameters.

What is missing?

Two things are missing and could be implemented on your own watch:

normalize the data of the datasets first add a bias term to our matrices

Consult this page for further information.

What have we learned?