Simple Neural Network Explanation: From Logistic Regression to Neural Network — Part 1
By: Esraa Madi
Tags: Supervised Learning, Classification, Neural Networks, Logistic Regression

Understanding Classification Learning, especially Logistic Regression, will help us lay down some major concepts that are fundamental to Neural Networks and its computationally intensive processes.

In this series, we’ll go through an overview of Supervised Learning then dive deeper with basic examples into how and when we could use both of Logistic Regression model and the Neural Network and what main differences are between them.

To keep this guide digestible and easy to read, this series is divided into four parts:

Part one: Introduction to supervised learning and Logistic Regression.

Introduction to supervised learning and Logistic Regression. Part two : Introduction to Neural Networks explains when to move from Logistic Regression to Neural network.

Introduction to Neural Networks explains when to move from Logistic Regression to Neural network. Part three : Train your Neural Network model — Feedforward propagation.

Train your Neural Network model — Feedforward propagation. Part Four: Train your Neural Network model — Backward propagation. (coming soon)

If you’re familiar with any topic, you can skip to the next part.

So let’s get started! 😉

Index:

Supervised Machine Learning Supervised Learning: Classification Classification: Logistic Regression

1. Supervised Machine Learning:

Supervised machine learning is a type of machine learning algorithm that requires the data used in training a model to be labeled with correct answers. In other words, supervised learning algorithms use training data that has a known target either numbers or labels.

We use this data, called training data, to train a model to estimate or predict the desired known target. The training data includes input features (X) and target column (Y), so a supervised learning algorithm can predict (Y) column using those (X) features/columns.

There are two types of supervised machine learning algorithms :

Regression Classification

Regression: When the known target to be predicted is a continuous number. For example, in a regression problem, a model will learn to estimate a total student score after being trained on a dataset of students’ information shown in figure 1.

Figure 1: Regression Problem

We can break down the regression problem above as following:

Training set:

It is a dataset of students’ performance and some additional personal information that can be useful in predicting expected students’ final scores.

Model input (Features):

Some selected columns from the training set, such as student performance information include students’ grades on selected subjects ( math, sport, science) and personal information includes students’ age and attendance.

Model output (Target):

Total student score.

Classification: When the known target to be predicted is a label. For example, in a classification problem, the model will learn to identify whether a student passed or not after being trained on the same dataset of students’ information.

Figure 2: Classification Problem

We can break down the classification problem above as following:

Training set:

It is the same dataset of students’ information which is used here in predicting whether a student would pass or not. Model input (Features):

Same student performance information above includes students’ grades and personal information. Model output (Target):

Student has passed or not (0 or 1) or we can use any two labels, e.g. {True, False}, {‘fail’, ‘pass’}.

Figure 3: Classification Vs Regression

To summarize, the main difference between Regression and Classification is the output (target) we are trying to estimate. In Regression, the output is numerical /continuous while for Classification it is categorical/discrete.

Supervised Learning: Classification in Depth

Generally speaking, there are two main types of classification problems: Binary classification and Multi-class classification

Binary classification: When trying to predict a target that has two only possible values. For example, you can predict whether or not a student will pass {0,1}, predict if an email is spam or not {spam, not spam}, or predict employees churn’s {leave, stay}, etc.

Let’s try to explain this visually. In all charts below, we assume we have the same input features for the model (X1, X2) with a different predicted target each time depending on the problem we are trying to solve.

Figure 4: Binary classification

Multi-class classification: When trying to predict a target that has more than two possible values. For example, predict a flower’s color {red, purple, yellow}, predict a student’s grade {A, B, C, D}, or predict a passenger’s class {1st, business, regular}, etc.

In the following figure, we used the same input features before for the model (X1, X2) with a different predicted target each time.

Figure 5: Multi-class classification

The main function of any classification model is to take a dataset (containing N rows/observations) then finding the best observations separator based on given different classes. This separator could be a line, curve or any shape that can separate each class’s elements as much as possible. In the flower color example used above, we have 3 classes, {red, yellow, purple}, to solve this classification problem we need to find a good line (or any separator) that can classify these 3 classes {class1: red points, class2: yellow points, class3: purple points} into separate groups as shown below:

Figure 6: Line separator for multi-classes

Classification: Logistic Regression

Now that we understand what a classification problem is, let’s explore common algorithms used to solve it.

Logistic Regression is a classification algorithm that uses Linear Regression concept to classify the target variable, using a line separator (decision boundary).

To make this more clear, let’s consider the example where we want to predict whether an employee would leave the company or not (binary classification) given 2 features: vacation days (X1) and age (X2).

Figure 7: Training dataset

Let’s start building a Logistic Regression model step by step.

First, we want to apply the Linear Regression concept on our dataset (if you need a refresher on the concept, read this article). The objective of linear regression is to find a relationship between the target (whether or not an employee leaves the company) and selected features (age) and (vacation days). Using this relationship we can predict whether an employee would leave the company given his age and vacation days.

Figure 8: Training dataset plot

Above we just plot the target (Y) with one feature (Age) to keep it a 2D plot 😌, As we can see, all observations with different ages lie only on 2 values on the target:

0 (not leaving the company).

1 (leaving company).

To find a relationship between features and the target variable, we need to find the best fit line for the training dataset, which aims to minimize the distance between the predicted value and actual value of the target.

Figure 9: Linear regression model

The general equation of the fitted blue line is:

Y = a (X1) + b (X2)+ c

After applying our features and target, the line equation is:

employee_leaves_comp = a (vacation days) + b (age)+ c

Second, the equation above presents our Linear Regression model that has been trained using 2 features from our training dataset. So by applying any given (vacation days) and (age), we get Y value which should be 0 or 1 refers to if an employee will leave or not. Mathematically speaking, Y could hold a value other than 0 or 1. Therefore, to use this model in our prediction problem, we need to choose a point on the Y-axis to be as a threshold that classifies any point to one of the given classes {leave, not leave }. Usually, the natural threshold point is used which is 0.5.

Finally, we can summarize the simple prediction scenario for any given age and vacation days as:

Get Y value by mapping age and vacation days in the regression equation above. Compare calculated Y value with the threshold point, if :

{

Y ≥ 0.5, employee will leave the company.

Y < 0.5, employee will not leave the company.

}

That’s all we need; we have done the job using Linear Regression. It is a good fit, but what if we added a new data point (employee) to the above training dataset to show a more realistic picture of what’s going on 🧐

Figure 10: Linear regression model

As we can see, the line we fitted before doesn’t separate the points well anymore. It put some points (employees) who left the company in the group who’s not leaving. It seems now the linear regression is not enough to decide the class for given points 🙁. Just one additional employee (point) is disturbing the whole Linear Regression predictions. Hence we can use a smarter algorithm, Logistic Regression.

Unlike linear regression which output is unbounded and could be any numerical value, logistic regression has limited output value strictly ranges from 0 to 1. It transforms the output value using the logistic sigmoid function to return a probability value of it belonging in either class.

Sigmoid function maps any real value into another value between 0 and 1. We use it here to map predictions (Y) to probabilities. It looks like as follows:

Figure 11: Sigmoid function

It is defined by the formula:

To see how logistic regression works, let’s consider the same linear regression model equation.

Y = a (X1) + b (X2)+ c

As mentioned above, we use this equation to get a prediction (predicted Y) for any given employee’s age and vacation days. Then we will use sigmoid function to turn predicted Y to a value (Z) between 0 and 1.

Z value represents the probability of an employee belonging in either class {leave, not leave}.

Then we use the same threshold point rule above to compare probability (Z) with 0.5. If Z is larger than 0.5, then the employee will leave the company, otherwise he/she will not.

We mentioned earlier that the main function for any classification is finding the best separator (decision boundary) that can separate given observations to their classes as much as possible. For logistic regression, the separator is a line.

Figure 12: Decision boundaries for binary classes

The above figure shows our observation points using our two features only. The decision boundary intersects with first feature X1 in point L1 and second feature X2 in point L2. These points can be mathematically written as follows:

Starting with Linear Regression:

Assumed some values for a, b, c to keep calculation simple

Apply sigmoid function on the predicted value Y to get the probability.

Compare probability Z with the threshold value.

Let’s solve the equation:

Using the final equation for the decision boundary line:

Get L1 by substituting vacationDays = 0 in the equation , L1 is employee age when its vacation days equal 0.

in the equation L1 is employee age when its vacation days equal 0. Get L2 by substituting age = 0 in the equation, L2 is employee vacation days when its age equal 0.

After all these steps we got the logistic regression model that fits the training dataset better than linear regression by applying sigmoid function on the prediction. 💪🏽