A logistic regression from scratch
By: Dennis Bakhuis
Tags: Artificial Intelligence, Machine Learning, Deep Learning, Neural Networks, Data Science

A logistic regression from scratch

Translate Tensorflow to plain Numpy

In my previous blog post we worked on artificial neural networks and developed a class to build networks with arbitrary numbers of layers and neurons. While the blog referenced to the previous notebook explaining the prerequisites, there was not yet the accompanying article, which is this blog post.

This post is also available as a Jupyter Notebook on my Github, so you can code along while reading.

If you are new to Python and Jupyter, here is a short explanation on how I manage my Python environment and packages.

A short overview of the topics we will be discussing:

Link between neural networks and logistic regression One step back: linear regression From linear to (binary) logistic regression Round up

Link between neural network and logistic regression

When we hear or read about deep learning we generally mean the sub-field of machine learning using artificial neural networks (ANN). These computing systems are quite successful in solving complex problems in various fields, examples are, image recognition, language modelling, and speech recognition. While the name ANN implies that they are related to the inner workings of our brain, the truth is that they mainly share some terminology. An ANN generally consists of multiple interconnected layers, which on itself are build using neurons (also called nodes). An example is shown in figure 1.

Figure 1: Example of a typical neural network, borrowed from my follow up article on neural networks.

In this example, we have one input layer, consisting of four individual inputs nodes. This input layer is ‘fully connected’ to the first hidden layer, i.e. Fully connected means that each input is connected to each node. The first hidden layer is again fully connected to another ‘hidden’ layer. The term hidden indicates that we are not directly interact with these layers and these are kind of obscured to the user. The second hidden layer is on its turn fully connected two the final output layer, which consists of two nodes. So in this example we feed the model four inputs and we will receive two outputs.

Let’s now focus on a single neuron from our previous example. This neuron still is connected to all inputs, also called features. Using these features, the neuron calculates a single response (or output). A diagram of such a system is depicted in figure 2.

Figure 2: single neuron with four input features. The neuron has two operations: a linear part and the activation function.

The input features are named 𝑓¹, 𝑓², 𝑓³, and 𝑓⁴ and are all connected to the single neuron. This neuron executes two operations. The first operation is a multiplication of the features with a weight vector 𝑊 and adding the bias term 𝑏. The second operation is a so called activation function, indicated here by 𝜎. The output of the neuron is a probability between zero and one. The single neuron acts like a small logistic regression model and therefore, an ANN can be seen as a bunch of interconnected logistic regression models stacked together. While this idea is pretty neat, the underlying truth is a bit more subtle. There are many different architectures for ANNs and they can use various building blocks that act quite different than in this example.

The linear operation in our single neuron is nothing more than a linear regression. Therefore, to understand logistic regression, the first step is to have an idea how linear regression works. The next section will show a step by step example as a recap.

One step back: linear regression

What is a linear regression again?

Linear regression in its simplest form (also called simple linear regression), models a single dependent variable 𝑦 using a single independent variable 𝑥. This may sound daunting, but was this means is that we want to solve the following equation:

𝑦 = a𝑥 + 𝑏

In the context of machine learning, 𝑥 represents our input data, 𝑦 represents our output data, and by solving we mean to find the best weights (or parameters), represented by 𝑤 and 𝑏 in the linear regression equation. A computer can help us find the best values for 𝑤 and 𝑏, to have the closest match for 𝑦 using the input variable 𝑥.

For the next examples, let us define the following values for x and y:

𝑥 = [−2,−1,0,1,2,3,4,5]

𝑦 = [−3,−1,1,3,5,7,9,11]

The values for 𝑥 and 𝑦 have a linear relation so we can use linear regression to find (or fit) the best weights and solve the problem. Maybe, by staring long enough at these values, we can discover the relation, however it is much easier to use a computer to find the answer.

If you have stared long enough or just want to know the answer, the relation between 𝑥 and 𝑦 is the following:

𝑦 = 2𝑥 + 1

In the next section we will use Tensorflow to create our single neuron model and try to ‘solve’ the equation.

An implementation in Tensorflow

Before we start with Tensorflow, we should first organize our input data (𝑥) and output data (𝑦). For this we are going to use Numpy:

In the next step, we will import Tensorflow. It is always a good practice to check which version we are using:

Now we can create a model using Keras, which is now a part of Tensorflow. To do this, we will use the Sequential class, which can stack various layers ‘sequentially’ after each other. We use the Dense class from Keras to create a ‘fully connected’ layer, which consists of a single neuron (unit).

The Dense function is used to create layers of many fully connected neurons (logistic units). The parameter units is used to set the amount of neurons. We only use a single unit and therefore, we will set it to one. As this is the first ‘layer’ of our model, we need to tell Tensorflow what shape it can expect as an input. This is only necessary for the first layer.

Now that we have defined the model, we need to use the Compile() method to configure the model for training. The method requires at least two parameters, a loss function and an optimizer. The loss function is a measure for how well the model predicts the actual value. For this example we will use the mean squared error (average of the squared difference between the predicted and the actual value of 𝑦). The ‘learning algorithm’ will try to minimize the loss my adjusting (optimizing) the parameters (weights and bias) for each step. The optimizer defines a method to perform this Optimizing step and a common method is Gradient Descent, or in our case Stochastic Gradient Descent (SGD). This method will become more clear in the next section where we will implement it in plain Numpy.

Next, we use the Fit() method to let the algorithm learn the best parameters. While Tensorflow has many smart ways to address this problem, the recipe is more or less like this:

calculate the prediction ŷ using the current parameters (𝑊 and 𝑏) calculate the loss of the current values calculate the gradients of the loss function with respect to the parameters adjust the weights (optimize) using the gradients. repeat for the number of epochs, i.e. the number of times to go through the provided examples (dataset).

Do not worry to much if this is not completely clear. We will code each step in plain Numpy, with a thorough derivation of all the math used in the next section.

We have trained the model in Tensorflow using our training data and we are ready to give it a spin. Now we can use our model to ‘predict’ values it has never seen. This is sometimes also called inference. Let’s try the value of 12. We know that it should be 25.

Why is the value not exactly 25?

The model calculates the difference between the actual value and the predicted value and creeps slowly towards the actual value. Running the fit method for a longer time will get you closer to 25.

This was not that hard, but it might feel like some dark Jedi power. Therefore, in the next section we will implement this algorithm in plain Python (with the help of Numpy).

What actually is happening ‘under the hood’

In the previous section we gave a rough overview what Tensorflow is doing under the hood:

calculate the prediction ŷ using the current parameters (𝑊 and 𝑏) calculate the loss of the current values calculate the gradients of the loss function with respect to the parameters adjust the weights (optimize) using the gradients. repeat for the number of epochs, i.e. the number of times to go through the provided examples (dataset).

We will now implement exactly this in plain Python and hopefully come to a similar result as Tensorflow.

First we define the model parameters. These are the weights 𝑊, which is just a single value, because we only have a single input. Also we need to define the bias term 𝑏.

This are all the trainable parameters in our model, a single scalar for the weight and a single scalar for the bias.

Next we will define a function that makes a prediction using our current model parameters. In deep learning terms, this is called a forward pass. The variable of the predicted value is generally name ŷ (or A but you can learn this in my next article).

The function is named forward and uses the input vector 𝑋 and multiplies it with the weight parameter 𝑊 and adds the bias term 𝑏. Exactly as we described in the before-mentioned equation.

We can now test the function with the current parameters. Again, we will input a value of 12.0 but of course, it will return gibberish as the weights are randomly initialized.

-0.2122614846691141

Indeed, -0.212261… is not really what we were looking for, but we still have to train our model before it can make proper predictions. Before we can do that we need to calculate the current loss. As a loss we used the mean squared error of the predicted value ŷ and the actual value 𝑦.

Hopefully the math does not scare you, but if you take the time, it is not that hard. The variable 𝑚 here is the amount of examples (points in the dataset). Our 𝑋 holds eight values and therefore, 𝑚=8. When we have a 1/𝑚 followed by as sum (∑) over all 𝑚 values, it is nothing more than an average of the values that are 𝑖𝑛𝑠𝑖𝑑𝑒 the summation. Here, we take the average over all (𝑦−ŷ)² which is the difference between the true value 𝑦 and the predicted value ŷ squared. The square is important because a negative difference and a positive difference would cancel each other out if we would not square the difference. Now that we fully understand the mean squared error, we can implement it in code:

To see what the loss is between our previously calculated value, we can do this:

149.13933056993267

As we can see, the loss is pretty large and our weights are quite off. Therefore, we need to update our trainable parameters to make better predictions. To do so, we need to first calculate the gradients 𝛿Loss/𝛿𝑊 and 𝛿Loss/𝛿𝑏. Maybe your differential skills are a bit rusty. The trick is to apply the chain rule. Another benefit is that the average operator (the sum) is linear and therefore, we can ignore it in the differential, and put it back later.

To calculate 𝛿Loss/𝛿𝑏 we only need to repeat the last step, with respect to 𝑏:

We will implement this in the ‘backward pass’ function. As variable names get a bit long and we are a bit lazy, we will just call them 𝑑𝑊 and 𝑑𝑏.

So now we can get the gradient of our previous test example:

(-605.0942756320587, -50.42452296933823)

The last function we need, before we can compose our training loop is the update function. This function will ‘optimize’ our weights one step. This is the actual gradient descent in which we subtract (descent) the gradient from our current weights. Gradient descent is defined as follows:

Here we have a new parameter 𝛼 which is called the learning rate. This value will set the speed in which we try to converge to the minimum loss. For this example we will set it to 0.01. There is no golden value for the learning rate and this value needs to be fine-tuned for each problem. Our code for the update function is as follows:

To update our current model parameters we simply do:

(array([6.0332543]), 0.5042452296933823)

Alright, we have updated our weights for the first time. To improve the weights, we have to repeat this process many times. For this we will write a loop:

129.7037237445872

0.06819476739792682

0.00458155318441586

0.0003078050915820409

2.067944441333496e-05

We are in luck and the loss, i.e. the difference between our model prediction and the actual value, is decreasing. How would we now predict, when we input a value of 12.0?

array([25.00101652])

I hope that this dark magic in Tensorflow is now a bit more clear. In the next section we will use our newly acquired knowledge to solve a binary logistic regression problem.

From linear to (binary) logistic regression

What is the difference exactly?

The differences between a linear regression and a logistic regression are not that major. There are two differences from the previous code we created. First, our linear regression model only had a single feature, which we inputted with 𝑥, meaning that we only had a single weight. In logistic regression, you generally input more than one feature, and each will have its own weight. Technically, you could have a single feature, but it would be nothing more than an if-statement (think about it). Increasing the number of features will change the previous simple multiplication to a matrix multiplication (dot product). Secondly, we will add a so called activation function to map this value between 0 or 1. Let’s remind ourselves again of our simple model:

By convention in Tensorflow (from what I have understood), the input vector has columns for features, and rows for examples. If we would have 2 datapoints the input matrix would look like this:

The superscript shows the feature number, the subscript indicates the example.

Each of these inputs are associated with their own weights. The node itself has two distinct operations. The first is the dot product between the weights vector and the input vector. The second is the Sigmoid function. The weight vector 𝑊 in this example has four weights:

In the node we first compute the linear part:

For our example this will look like this:

Notice that the result is only a single value for each example.

For all values of 𝑥, which can be from −∞ to +∞ (all real numbers), the Sigmoid function maps 𝑥 between 0 and 1. Values for 𝑥 close to zero have the largest effect, as these are in the ‘linear regime’. Very large, or very small are only clipped to 1 and 0 respectively. The mathematical definition of the Sigmoid function is:

This is all there is to the logistic unit. The Sigmoid function gives the node its non-linear character. Many of these units together can do almost magical things. In the next section we will first make a logistic regression model in Tensorflow.

An implementation in Tensorflow

Before we can start we first need some data to do a logistic regression. I downloaded the titantic dataset from Azeem Bootwala from Kaggle to have a play for this example. It can be downloaded from here:

https://www.kaggle.com/azeembootwala/titanic

(792, 17)

First, always inspect the columns and data types:

<class 'pandas.core.frame.DataFrame'>

RangeIndex: 792 entries, 0 to 791

Data columns (total 17 columns):

# Column Non-Null Count Dtype

--- ------ -------------- -----

0 Unnamed: 0 792 non-null int64

1 PassengerId 792 non-null int64

2 Survived 792 non-null int64

3 Sex 792 non-null int64

4 Age 792 non-null float64

5 Fare 792 non-null float64

6 Pclass_1 792 non-null int64

7 Pclass_2 792 non-null int64

8 Pclass_3 792 non-null int64

9 Family_size 792 non-null float64

10 Title_1 792 non-null int64

11 Title_2 792 non-null int64

12 Title_3 792 non-null int64

13 Title_4 792 non-null int64

14 Emb_1 792 non-null int64

15 Emb_2 792 non-null int64

16 Emb_3 792 non-null int64

dtypes: float64(3), int64(14)

memory usage: 105.3 KB

I expect that Azeem did not save the set using index=False option and therefore we have a ‘Unnamed: 0’ column. This one is redundant with our current index so we can remove it. Also the PassengerId is not very useful for our model, let us remove that column too. After that, let us sample the dataset to get an idea:

Azeem already did some preprocessing. The target variable 𝑌 is ‘Survived’, all other columns are features. A short description of the features:

- Sex: 0 or 1 -> male or female

- age: value rescaled between 0 and 1

- fare: ticket price rescaled between 0 and 1

- Pclass_1 .. Pclass_3: One-hot encoded Passenger class

- family size: rescaled value between 0 and 1 of family size.

- title_1 .. title_4: mr, mrs, master, miss one-hot encoded

- emb_1 .. emb_3: Embark location one-hot encoded.

In total we will have 14 features.

For this example, the data will suffice and I will not go into detail on how this data has become what it is. Honestly, I do not know myself and have just downloaded it from Kaggle ;-).

Lets put these variables in the format we defined before (𝑋 and 𝑌). Here 𝑌 corresponds to the label if a person has survived. We have a total of 792 examples. Therefore, the shape for 𝑌 is (𝑚,1) where 𝑚 = 792. For 𝑋 we expect (𝑚, 14), where the columns are the features.

((792, 14), (792,))

Now that we have prepared the data we can create a model in Tensorflow:

The model in Tensorflow is very similar to our linear regression model. The input has changed from 1 to 14 features and we added the Sigmoid activation function. Next, we must again compile our model:

In this example I changed the loss into ‘binary_cross_entropy’. This is another loss function which works better for binary logistic regression problems. If you are interested in the inner workings I recommend wikipedia.

We added an additional metric called ‘accuracy’ which is now calculated for each epoch. Now we are ready to train our model:

Well, these were all steps required for training a binary logistic classifier in Tensorflow. We achieved an accuracy of ~80% which is not too bad for the effort we put in.

We can extract the weights 𝑊 and the bias 𝑏 from our model. These values we can later compare to our own implementation of the logistic regression:

Plotting the loss is also quite easy:

Don’t mind the nice XKCD wobble :-). It is quite impressive how few steps are required to get to such a result. In the final section we will unveil the Dark Jedi arts being performed by Tensorflow.

What is actually happening?

Well, the general recipe has not changed:

forward pass calculate loss backward pass update weights repeat

The steps itself, need minor modification. The forward pass will be the more general dot product and we need to add the activation function. The loss function is binary cross entropy, which is of course different from the mean square error. The backward pass will calculate the gradient of the new loss function with respect to 𝑊 and 𝑏. As we now also have an activation function, we will have an additional step. The update-weights function is unchanged, the final loop is also very similar.

In Numpy, and in math in general as far as I know, the dot product needs the shapes of the vectors (and matrices) to be compatible:

This means that the number of columns of 𝑋 must be equal to the number of rows in 𝑌. To make ourselves a bit easier, we will Transpose our input vector 𝑋 and flip the vector. This will result that the rows will be features and the columns will be examples.

(14, 792)

For testing, let us just select two examples, to make it a bit more readable:

(14, 2)

Lets define our weights. As we have 14 features, our vector 𝑊 will have 14 values. The bias is a constant for the whole node, and is only a single value.

(14, 1)

Next, we need to define the Sigmoid function:

array([3.72007598e-44, 5.00000000e-01, 5.24979187e-01, 1.00000000e+00])

Now lets redefine our forward function, and make it use the dot product and the activation function. We can split these in two steps:

𝑍=𝑊𝑋+𝑏

A = 𝜎(𝑍)

Note that 𝑊𝑋 is a dot product.

array([[0.48951622, 0.50149394]])

Now that we have actual predictions, we can write our loss function to measure how well our predictions are. This is done with the binary cross entropy, for which I will simply give the equation:

It might happen we try to calculate a log(0) which is of course not defined. To avoid the warning, we will add a tiny value to our loss. As it is super small, the difference is not noticeable, but does help suppress the warning. One less warning a day keeps the ….

1.3625601508159457

Next is the backwards pass. For this, We would need to differentiate the Loss function with 𝑊 and 𝑏. Not to bore you guys, I have provided these functions, but I will not stop you from calculating the differentials yourself:

In Python, this looks like this:

(array([[ 0.06118953],

[-0.01277168],

[-0.0078041 ],

[-0.06231326],

[ 0. ],

[ 0.06118953],

[-0.00011237],

[-0.00112373],

[ 0. ],

[ 0. ],

[ 0. ],

[-0.06231326],

[ 0. ],

[ 0.06118953]]),

-0.0011237299781017493)

Almost there, next we need to update the weights. The function has not changed from our previous example:

(array([[-0.01830035],

[ 0.00088324],

[-0.01122826],

[-0.00589117],

[-0.00893116],

[-0.01335291],

[-0.00061042],

[ 0.00065638],

[ 0.00410113],

[-0.00572882],

[-0.00801334],

[ 0.01374348],

[ 0.01274699],

[-0.01275547]]),

1.1237299781017494e-05)

To compare the results, we can calculate the accuracy. However our activation function returns a probability between 0 and 1. By definition, values <= 0.5 are rounded to 0 and values > 0.5 are rounded to 1. This is slightly different from the regular round function so we will make our own function for this:

array([[0, 1]], dtype=uint8)

Now we have everything to create our training loop. We will store some metrics to plot afterwards:

loss: 17.16646271874957 accuracy: 64.4%

loss: 12.757526841623513 accuracy: 75.9%

loss: 11.688361064742441 accuracy: 79.5%

loss: 11.150776615503466 accuracy: 79.5%

loss: 10.83231532525657 accuracy: 79.8%

loss: 10.622380143288218 accuracy: 79.8%

loss: 10.473049466548982 accuracy: 79.8%

loss: 10.36043263717339 accuracy: 80.8%

We trained the network and got a final accuracy of just above 80%, very similar results as when using Tensorflow. Of course, with a bit more effort, but also with a bit more fun. Let’s plot our metrics:

Of course, we have calculated the metrics on the training data. To get a proper idea on the model, this should be done using the test data. This, I leave open to the reader ;-).

Round up

Well, this was it for the tutorial on a Logistic regression. Hopefully you got an idea on how Logistic regression work and that Tensorflow is not only black magic. I found it a great exercise to write these from scratch and as you have seen, it is not that difficult either.

Please let me know if you have any comments or suggestions.