Simplified Mathematics behind Neural Networks
By: Shubham Dhingra
Tags: Gradient Descent, Data Science, Neural Networks, Artificial Intelligence, Machine Learning

Neural Networks are universal approximators that maps data to information. What does this mean? Can Neural Networks solve any problem? Neural Networks are a proven solution for scene-by-scene/frame-by-frame analysis, stock price prediction, in retail, and for many other purposes. Many of us are using it at the enterprise level, but how many of us truly understand it?

To answer the question, ‘Can Neural Networks solve any problem?’, let’s take it from the basics. A NeuralNet is made up of vertically stacked components called layers: input, hidden, and output. Each layer consist of a certain number of neurons. The input layer has attributes (features) of the dataset. There can be multiple hidden layers with multiple neurons and the output layer can have multiple neurons based on the problem statement.

Understanding Perceptron and Activation functions

Perceptron (or a neuron) is a fundamental particle of neural networks. It works on the principle of thresholding. Let f(x) be a summation function with a threshold of 40.

Fig 1. Firing of Neurone (Image by Author)

In both cases, the defined function is returning the addition of two inputs, x₁, and x₂. In case 1, the function returns 30 which is lesser than the threshold value. In case 2, the function returns 50 which is greater than the threshold and the neuron will fire. Now, this function gets complicated than this. A neuron of a typical neural network receives a sum of input values multiplied by its weights and added bias, and the function, also known as the Activation function or Step function, helps in making a decision.

Fig 2. Perceptron (Image by Author)

Activation Functions convert the output from the node into the binary output; 1 if the weighted input crosses the threshold, 0 otherwise (depends on the activation function). There are three most commonly known activation functions:

Sigmoid

Sigmoid is a widely used activation function that helps in capturing non-linear relationships.

Fig 3. Sigmoid Curve (source)

For any value of z, function Φ(z) will always return a binary (0/1) output. Because of this, it is widely used in probability-based questions.

Tanh (Tangent Hyperbolic)

It is more-or-less like Sigmoid function but tanh ranges from -1 to 1 which makes it suitable for classification problems. It is non-linear.

Fig 4. tanh curve (Image by Author)

ReLu (Rectified Linear Unit)

It is the most used activation function in deep learning because it is less complex than other activation functions. f(x) returns either 0 or x.

Fig 5. ReLu curve (Image by Author)

This makes computation easy as the derivative of the ReLu function returns either 0 or 1.

NeuralNet

To understand the black box of a Neural Network, let us consider a basic structure with 3 Layers; an Input layer, a Dense/Hidden layer (connected on both sides of the neurons), and an Output layer.

Fig 6. A simple Neural Network (Image by Author)

Weights and biases are randomly initialized. The accuracy of the output of a neural network is all about finding the optimal values for weights and biases by continuously updating them. Let us consider an equation, y = wx where ‘w’ is the weight parameter and ‘x’ is the input feature. In simple terms, weight defines the weightage given to the particular input attribute (feature). Now, the solution of the equation y = wx will always pass through the origin. Hence, an intercept is added to provide freedom so as to accommodate the perfect fit which is known as the bias and the equation becomes ŷ = wx + b, which we are all familiar with. Hence, bias allows activation functions’ curve to adjust up-or-down the axis.

Let us now see how complicated a neural network can get. For our network, there are two neurons in the input layer, four in the dense and one in the output. Every input value is associated with its weights and biases. The combination of input features with weights and biases goes through the dense layer where the network learns the feature with the help of the activation function and it has its own weights and biases, and finally makes the prediction (output). This is the Forward Propagation. So, how many total parameters are there for our network?

Fig 7. Total Parameter calculation of a Neural Network (Image by Author)

For such a simple network, there are a total of 17 parameters that are needed to optimize in order to get an optimal solution. As the number of hidden layers and the number of neurons in it increases, the network gains more power (up to a certain point), but then we have an exponential number of parameters to optimize which might end up taking a huge amount of computing resources. So, there is a trade-off.

UPDATING THE NETWORK

After a single iteration of forward propagation, Error is calculated by taking the (squared) difference between actual output and predicted output. In a network, inputs and activation functions are fixed. Hence, we can change the weights and biases to minimize the error. The error can be minimized by noticing two things: change in error by changing weights by small amounts, and the direction of the change.

Cost Function

A simple neural network predicts value based on Linear Relationship, ŷ = wx + b, where ŷ (predicted) is the approximation of y (actual). Now, there can be several linear lines fitting ŷ. To choose the best fit line, we define the Cost Function.

Let ŷ = θ₀ + xθ₁. We need to find values of θ₀ and θ₁ such that ŷ is as close to the y. To do that, we need to find values of θ₀ and θ₁ such that the following defined error is minimum.