How The Artificial Neural Network Work-Part 2
By: Dilan Bakır
Tags: Gradient Descent, Artificial Intelligence, Perceptron Algorithm, Artificial Neural Network

In Part 1, we learned about the basic components of the artificial neural network. In this section, let’s talk about how an artificial neural network works and its types. Let’s code a Perceptron together. And let’s see how it works together. If you’re ready, let’s start :)

Types of Neural Networks

What Is a Perceptron?

A perceptron is a simple binary classification algorithm, proposed by Cornell scientist Frank Rosenblatt. It helps to divide a set of input signals into two parts — “yes” and “no”. But unlike many other classification algorithms, the perceptron was modeled after the essential unit of the human brain — the neuron and has an uncanny ability to learn and solve complex problems.

A simple perceptron model

Perceptron is a single layer neural network and a multi-layer perceptron is called Neural Networks. In this type of neural network, there are no hidden layers. It takes an input and calculates the weighted input for each node. Afterward, it uses an activation function (mostly a sigmoid function) for classification purposes.

Where we can use perceptrons?

Perceptrons’ use lies in many case scenarios. While a perceptron is mostly used for simple decision making, these can also come together in larger computer programs to solve more complex problems.

For instance:

Give access if a person is a faculty member and deny access if a person is a student. Provide entry for humans only. Implementation of logic gates

Steps involved in the implementation of a neural network:

A neural network executes in 2 steps :

1. Feedforward:

On a feedforward neural network, we have a set of input features and some random weights. Notice that in this case, we are taking random weights that we will optimize using backward propagation.

2. Backpropagation:

During backpropagation, we calculate the error between predicted output and target output and then use an algorithm (gradient descent) to update the weight values.

Why do we need backpropagation?

While designing a neural network, first, we need to train a model and assign specific weights to each of those inputs. That weight decides how vital is that feature for our prediction. The higher the weight, the greater the importance. However, initially, we do not know the specific weight required by those inputs. So what we do is, we assign some random weight to our inputs, and our model calculates the error in prediction. Thereafter, we update our weight values and rerun the code (backpropagation). After individual iterations, we can get lower error values and higher accuracy.

Summarizing an Artificial Neural Network:

Take inputs Add bias (if required) Assign random weights to input features Run the code for training. Find the error in prediction. Update the weight by gradient descent algorithm. Repeat the training phase with updated weights. Make predictions.

Flow chart for a simple neural network:

Artificial Neural Network (ANN) Basic Flow Chart

The training phase of a neural network:

Training phase of a neural network

The Perceptron Learning Rule

The perceptron consists of 4 parts.

Input values or One input layer Weights and Bias Net sum Activation Function

The Neural Networks work the same way as the perceptron. So, if you want to know how neural network works, learn how perceptron works.

Steps of Perceptron’s :

Initialize weight values and bias

Forward Propagate

Check the error

Backpropagate and Adjust weights and bias

Repeat for all training examples

We don’t have to design these networks. We could have learnt those weights and thresholds, by showing it the correct answers we want it to generate.

Let input x = ( x1, x2, .., xn) where each xi = 0 or 1. And let output y = 0 or 1.

Algorithm is:

Repeat forever:

Given input x = ( x1, x2, .., xn). Perceptron produces output y. We are told correct output O. If we had wrong answer, change wi’s and t, otherwise do nothing.

Weight change rule:

If output y=0, and “correct” output O=1, then y is not large enough, so reduce threshold and increase wi’s. This will increase the output. (Note inputs cannot be negative, so high positive weights means high positive summed input.) Similarly, if y=1, O=0, output y is too large, so increase threshold and reduce wi’s (where xi=1). This will decrease the output. If y=1, O=1, or y=0, O=0, no change in weights or thresholds.

Hence algorithm is:

Repeat forever:

Given input x =( x1, x2, .., xn). Perceptron produces output y. We are told correct output O. For all i:

wi := wi + C (O-y) xi t := t — C (O-y)

Results:

->where C is some (positive) learning rate.

->Note the threshold is learnt as well as the weights.

->If O=y there is no change in weights or thresholds.

->If xi=0 there is no change in wi

The sequence of exemplars presented may go like this

Perceptron Example:

Below is a simple perceptron model with four inputs and one output.

A simple perceptron model

A set of data

What we have here is the input values and their corresponding target output values. So what we are going to do, is assign some weight to the inputs and then calculate their predicted output values.

In this example we are going to calculate the output by the following formula:

For the sake of this example, we are going to take the bias value = 0 for simplicity of calculation.

a. Let’s take W = 3 and check the predicted output.

The output when W = 3

b. After we have found the value of predicted output for W=3, we are going to compare it with our target output, and by doing that, we can find the error in the prediction model. Keep in mind that our goal is to achieve minimum error and maximum accuracy for our model.

The error when W = 3

c. Notice that in the above calculation, there is an error in 3 out of 4 predictions. So we have to change the parameter values of our weight to set in low. Now we have two options:

Increase weight Decrease weight

First, we are going to increase the value of the weight and check whether it leads to a higher error rate or lower error rate. Here we increased the weight value by 1 and changed it to W = 4.

Output when W = 4

d. As we can see in the figure above, is that the error in prediction is increasing. So now we can conclude that increasing the weight value does not help us in reducing the error in prediction.

Error when W = 4

e. After we fail in increasing the weight value, we are going to decrease the value of weight for it. Furthermore, by doing that, we can see whether it helps or not.

Output when W = 2

f. Calculate the error in prediction. Here we can see that we have achieved the global minimum.

We can see that there is no error in prediction.

Now what we did here:

First, we have our input values and target output. Then we initialized some random value to W, and then we proceed further. Last, we calculated the error for in prediction for that weight value. Afterward, we updated the weight and predicted the output. After several trial and error epochs, we can reduce the error in prediction.

Illustrating our function

In real-life data, the situation can be a bit more complex. In the example above, we saw that we could try different weight values and get the minimum error manually. However, in real-life data, weight values are often decimal (non-integer). Therefore, we are going to use a gradient descent algorithm with a low learning rate so that we can try different weight values and obtain the best predictions from our model.

How To Implement The Perceptron Algorithm In python

Perceptron for the OR gate:

What is Stochastic Gradient Descent?

Gradient Descent is a machine learning algorithm that operates iteratively to find the optimal values for its parameters. It takes into account, user-defined learning rate, and initial parameter values.

Working: (Iterative)

1. Start with initial values.

2. Calculate cost.

3. Update values using the update function.

4. Returns minimized cost for our cost function

Generally, what we do is, we find the formula that gives us the optimal values for our parameter. However, in this algorithm, it finds the value by itself!

Implementation of a Neural Network In Python:

*Assign input values

*Target output

*Assign weights

*Adding Bias Values

*Assigning a Learning Rate

* Applying a Sigmoid Function

*Derivative of sigmoid function: (In gradient descent algorithm we are going to need the derivative of the sigmoid function)

The main logic for predicting output and updating the weight values:

Compilation result:

Check the Values of Weight and Bias:

Predicts: