Understanding a Neural Network through scratch coding in R; A novice guide
By: Abhibhav Sharma
Tags: Scratch Programming, Neural Network, Gradient Descent, Perceptron, Machine Learning

If you have ever delved into the world of data science, then it will not be an absurdity for me to assume that by now you have certainly encountered the term Neural Networks somewhere, at some point in your journey towards probing Machine Learning and Artificial Intelligence or data science in general.

Figure 1. A Neural Network with one hidden layer (Image by author)

A most approved definition for Neural Network (NN) is that it is a brain-inspired computer architecture containing varying network topologies of functions, where the nodes are interconnected in a specific fashion to unmask the underlying patterns in a dataset by following a series of algorithms iteratively (Figure 1). Perhaps, with a little mathematical maturity and a slight knowledge of optimization theory, it is criminally simple to call a Neural Network as a function approximator or a regressor. From my previous article, we saw through a practical example that how the high-level APIs such as Keras and Tensorflow makes it really simple to build and train a neural network. However, the simplicity provided by Keras and Tesnsorflow plagues a beginner with tenacious dependency and makes it really enigmatic for a novice to comprehend the actual underlying mathematics and reasoning behind the process of learning (Forwardfeeding-backpropagation). From my articles, you can probably tell that I am a very visual person, and when it comes to learning, I think that engaging visuals facilitates deeper comprehension. Here I will try to slightly unmask this black box and we will take a gentle dive into the mathematical formalism of Neural Network incorporating gradient descent optimizer and comprehensively try to build our own. Don’t worry If you are not comfortable with elementary linear algebra and differential equations as the visuals are enough to get the concept rolling in your brain console. And again, If you are comfortable with these mathematical prerequisites then you must appreciate how overhyped this model really is.

Now, let’s start!!

Neural Network Architecture

A simple NN architecture with one hidden layer is elucidated in Figure 2. The first layer is called the input layer. Each feature of the dataset act as a single neuron/node (red-colored). Given a tuple of weights, the linear combination of these nodes along with a bias (blue-colored) generates a single node of the next consequent layer called the hidden layer (yellow colored). Notice that the contribution of the iᵗʰ node of the input layer in the formation of the jᵗʰ node of hidden layer value hⱼ is the product of iᵗʰ node value Xᵢ and the iᵗʰ element of the jᵗʰ tuple of weight i.e wⱼᵢ. A similar scheme is then repeated for the hidden layer and the output layer.

Figure 2. Elucidating the Neural Network (Image by author)

At this point I recommend you try visualizing the weight tuples as column vectors, combining to form a matrix W of rows equals to the number of nodes in the current layer and the column equal to the number to nodes in the consequent layer. So for the given network in the animation, the weight matrix W corresponding to the input layer and hidden layer would be:

This kind of matrix apprehension of weights is important to comprehend the upcoming underlying mathematics.

Since we all know that the conceptualization of the NN is inspired by brain functioning and at this point, it is important to mention the synaptic ends of a brain cell/neuron and its analogy with a NN. Every neuron value generated by the above operation is analogous to a stimulus that a brain cell carries and is suppose to pass it to the next connected neuron. There is a mechanism in the synaptic end of the axons that decides whether the stimulus is powerful enough to be passed on to the dendrites of the next neuron. Analogously, this activation is carried out by an activation function in NN. So precisely, every generated node value is passed through an activation function, which decides whether to pass the stimulus and how much of the stimulus is to be passed.

Figure 3. A sigmoid function ranging from -inf to +inf (Image by author)

The most popular activation functions are Sigmoid, ReLU, Tanh, and Softmax (a generalized sigmoid). While the ReLU function permits only the positive values to be passed on, the sigmoid activation function, S(x) uniquely maps any real number x into the range (0,1) Figure 3. Mathematically, S(x)= 1/(1+eˣ). Interestingly, the differential of S(x) i.e dS(x)/dx is equals to S(x)(1-S(x)). This invites extreme mathematical convenience while calculating the gradient and a reason why sigmoid is the most celebrated activation function. This process of information transformation from one layer to another is called forward-feeding.

Once a final output layer is estimated, the performance is then evaluated using an error-index function P. Mathematically, P = ll d — zll²/2, where d is the actual/desired output and the z is the estimated output. The optimization objective is to minimize the P. There are numerous optimizers in practice such as the Newton Descent method, ADAM, etc. Here we will focus on the most popular and widely used gradient descent method.

Since P is a function of z, which in turn is a function of weights W and bias b. So to minimize the P, the change in weight, ▽W is evaluated in the direction where the partial of P with respect to weight dP(z, W, b)/dW = ▽P decreases (Figure 4).

Figure 4 (Image by author)

Now let's see how do we calculate these required gradients. So, If we map this prelude with our Figure 2 instance, the process looks like the below cartoon (Figure 5). Notice that to calculate the gradient of performance index with respect to the weights i.e dP/dW₁ and dP/dW₂, we applied chain rule starting from the right end of the process, propagating backward. (Recall that the performance index was P = ll d — zll²/2.). This procedure is called backpropagation.

Figure 5. Chain rule in Backpropagation (Image by author)

The gradient ▽P= Σ dPᵢ /dWᵢ for i = 1,2,3…,k+1 where k is the depth of architecture. Once the ▽P is evaluated, the weights and biases are upgraded respectively as:

W_new = W_current — η*(▽P) and b_new = b_current — η*(▽P)

Where the value η is the learning rate. These two equations are very important and we are frequently going to refer to them. With this jotting, we will now proceed to make our model step-by-step