Explained: Deep Learning in Tensorflow — Chapter 0 | by Sonu Sharma | Towards Data Science

I guess people like to read about the things that they want the most and experience the least.— Bernard Lowe (Westworld)
And guess what? embarking on this series. The series illustrates Deep Learning. ML / DL! Hmm. Actually, DL is not different from ML, but DL(Deep Learning) is a subfield of ML(Machine Learning) concerned with algorithms that are inspired by the structure and function of the brain known as Artificial Neural Network(ANN). As “neuron” is the basic working unit of the brain, it is also the basic unit of computation in a neural network.
We will straight forward discuss neuron here.
Without discussing the biological analogy of neurons, let’s understand artificial neurons directly. By the way, there are around approximately 60–100 billion neurons that can be found in the human nervous system and they are connected with approximately 1014 -1015 synapses. :) A neuron which is the basic building block of an artificial neural network receives the feature as input and produces the output after doing the computation which happens as follows:
Input Feature Vector (X): It is the characteristics of the input dataset which helps in drawing a conclusion about a certain behavior. It may be one hot-encoded, embeddings, etc.Weights and Biases (W & B): In general, weights, w1,w2,... are real numbers expressing the importance of the respective inputs to the output. Biases are an extra threshold value added to the output.Loss (L): Loss is the objective function that tells how close the prediction is with respect to the original outcome. It’s also known as cost function. The objective as a training mechanism is always to minimize the value of this cost function. In other words, we want to find a set of weights and biases which make the cost as small as possible. There are several loss functions as Mean Squared Error (MSE) popular among regression problems, categorical or binary cross-entropy popular among classification problems.Optimizers: These are used to minimize the loss function by updating the weights and biases. Stochastic Gradient Descent is a popular one. Here is a nice explanation for optimizers.Activation Function: Activation function decides, whether a neuron should be activated or not by calculating a weighted sum and further adding bias with it. But, why do we need activation function? The answer is: if we chain several linear transformations, all we get is a linear transformation. For example, if f(x) = 2x+3 and g(x) = 5x-1 in two neuron from adjacent layer. Then, chaining these two will give linear function i.e. f(g(x)) = 2(5x-1)+3. So, if we don’t have non-linearity between layers, then even a deep stack of layers is equivalent to a single layer. The purpose of the activation function is to introduce non-linearity into the output of a neuron. ReLU (Rectified-Linear Unit) is the most widely used activation function.Learning Rate(η): It is the rate at which the weights and biases should be changed in each update.
Input Feature Vector (X): It is the characteristics of the input dataset which helps in drawing a conclusion about a certain behavior. It may be one hot-encoded, embeddings, etc.
Weights and Biases (W & B): In general, weights, w1,w2,... are real numbers expressing the importance of the respective inputs to the output. Biases are an extra threshold value added to the output.
Loss (L): Loss is the objective function that tells how close the prediction is with respect to the original outcome. It’s also known as cost function. The objective as a training mechanism is always to minimize the value of this cost function. In other words, we want to find a set of weights and biases which make the cost as small as possible. There are several loss functions as Mean Squared Error (MSE) popular among regression problems, categorical or binary cross-entropy popular among classification problems.
Optimizers: These are used to minimize the loss function by updating the weights and biases. Stochastic Gradient Descent is a popular one. Here is a nice explanation for optimizers.
Activation Function: Activation function decides, whether a neuron should be activated or not by calculating a weighted sum and further adding bias with it. But, why do we need activation function? The answer is: if we chain several linear transformations, all we get is a linear transformation. For example, if f(x) = 2x+3 and g(x) = 5x-1 in two neuron from adjacent layer. Then, chaining these two will give linear function i.e. f(g(x)) = 2(5x-1)+3. So, if we don’t have non-linearity between layers, then even a deep stack of layers is equivalent to a single layer. The purpose of the activation function is to introduce non-linearity into the output of a neuron. ReLU (Rectified-Linear Unit) is the most widely used activation function.
Learning Rate(η): It is the rate at which the weights and biases should be changed in each update.
There are several ways of finding optimal LR through scheduling: Power, Icycle and exponential scheduling. One of the ways is to train the model for a few hundred iterations, exponentially increasing the LR from very small value to a very large value and hen looking at the learning curve picking a learning rate slightly lower than the one at which the learning curve starts shooting back up. As in the LR vs Loss curve on the left side, around 1/10 is the optimal LR.
“Essentially, all models are wrong, but some are useful.” — Box, George E. P.; Norman R. Draper (1987).
When more than one neurons are stacked over each other where connections between the units do not form a cycle, then it forms a single layer of dense neural network. When one or more than one layer is arranged parallel to each other (may have a different number of units of neurons) in which one being the input and output layer at the beginning and ending respectively, and another being the hidden layer in between these two layers such that information (output from each neuron) moves in only one direction, forward, from the input nodes, through the hidden nodes (if any) and to the output nodes, it is known as feed-forward neural network. If only the output layer is present in an ANN, then it is called single-layer FFNN and if it contains hidden as well as input and output layer, it is known as multi-layer FFNN.
The no. of parameters, means the total number of weights and biases are used throughout the neural network. Since at each layer, a matrix of weights is generated and updated whose dimension is Rm*n as explained above and biases are also added with each input. So, the total number of parameters will be the sum of weights and biases at each layer. The formula comes out to be:
If i is the number of inputs, H = {h1,h2,h3.. hn} are the number of hidden units and o is the number of outputs. Then,
For the example on the left side,
#model in Kerasmodel = Sequential()model.add(Input(3))model.add(Dense(5))model.add(Dense(2))
i=3 , o=2 and H = {5}
so, the total no. of parameters = (3*5 + 5*2) + (5+2) = 32
Backpropagation is about updating the weights and biases in the network after each forward pass of the information. After making a prediction (forward pass), it measures the error and goes through each layer in reverse to measure the error contribution from each connection (reverse pass) using chain rule and finally, tweaks the connection weights to reduce the error. The goal of backpropagation is to compute the partial derivatives ∂L/∂w and ∂L/∂b of the cost function L with respect to any weight w or bias b in the network. Since FFNN is a really complex architecture consists of a large number of neurons where linear regression kind of problem is solved neglecting the activation function having the equation as WX + B, so we will use a linear function of features as target function to understand the backpropagation.
Let the function estimated be ŷ and original function be y where a quadratic cost function (L):
Let’s calculate the gradient of L w.r.t. ŷ, w1, w2, and b
Now, we will know how to update the weights and bias using SGD as optimizer:
Let’s take a dataset and try to understand one iteration of it, then we will walk through the code :
x1 | 2  |  1   |  3  | -1   | -3x2 | 3  |  1   | -1  |  1   | -2y  |18  |  10  |  8  |  6   | -7
For the initialization, assume w1= 5, w2=8, b=-2, η = 0.02
After this parameter update, let’s see the change in the loss:
As we continue further and perform the calculation over all the iterations, we will be able to find the best fit close to the target function.
As we know, Tensorflow is a powerful library for numerical computation, particularly for large-scale Machine Learning developed by Google Brain Team. Let’s understand the basics of it. Tensorflow’s API revolves around tensors, which flows from operation to operation — and hence the name Tensorflow. Tensor is basically a multidimensional array, just like Numpy. We can define a constant tensor as follow:
tf.constant(2)#<tf.Tensor: id=14998, shape=(), dtype=int32, numpy=2>tf.constant([[1.,2.],[3.,4.]]) # <tf.Tensor: id=14999, shape=(2, 2), dtype=float32, numpy=array([[1., 2.],       [3., 4.]], dtype=float32)>
We can perform several operations on the tensors such as square, sum, transpose, etc. But, here tf.constant defined tensors are immutable and we can’t modify it. So, we need tf.Variable in that case:
#<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=2.0>a=tf.Variable(2.0)# a values changes to 3.0a.assign(3.0)#add and subtract the tensor by given valuea.assign_add(2.0) # a => 5.0a.assign_sub(1.0) # a=> 4.0
Let’s look at how we can compute the gradients automatically using Autodiff so as to solve the problem discussed above. We need to create a context of tf.GradientTape will automatically record every operation and finally, it can tell the gradients.
with tf.GradientTape(persistent=True) as tape:      #initializing the loss function with next set of training set            loss = loss_function(y,x1,x2)             # finding the gradient of loss function w.r.t. w1,w2 and b      gradients = tape.gradient(loss, [w1,w2,b])
Here, gradients give the partial derivatives ∂L/∂w and ∂L/∂b which we need to update the weights and biases with, by multiplying it with the factor of L.R. Let’s see how we can do so:
We have defined the loss_function which takes the training set as input followed by finding the gradients of loss function w.r.t weights and biases. The update_weight method updates the new weight after each iteration. As we can see, the new weights and biases are the same as we calculated manually above in the backpropagation section. We run it for 7 iterations and the values of weights and biases are getting closer to the actual ones which are [w1=2,w2=3,b=5].
TF2.0 brings tf.function decorator which transforms a subset of Python syntax into portable, high-performance TensorFlow graphs. If it is annotated over a method, the Python control flow statements and AutoGraph will convert them into appropriate TensorFlow ops. For example, if statements will be converted into tf.cond() if they depend on a Tensor . After annotation, the function will be like any other method in python, but it will help in getting benefits of eager execution. For example:
@tf.functiondef simple_nn_layer(x, y):    return tf.nn.relu(tf.matmul(x, y))x = tf.constant([[-2,-1],[3,4]])y = tf.constant([[3,4],[1,2]])simple_nn_layer(x, y)<tf.Tensor: id=15133, shape=(2, 2), dtype=int32, numpy=array([[ 0,  0],       [13, 20]], dtype=int32)>
Using together gradientTape and tf.function, we can get benefitted in terms of debugging as well. We can see how parameters are changing, accuracy is getting affected in each iteration. Keras models can also be used in AutoGraph code for this purpose.
Following up, do visit Explained: Deep Learning in Tensorflow — Chapter 1: About training the ANN and data preprocessing. Keep watching the space for more updates.
neuralnetworksanddeeplearning.com
towardsdatascience.com
https://towardsdatascience.com/counting-no-of-parameters-in-deep-learning-models-by-hand-8f1716241889
Book: Hands-on Machine Learning by Aurelien Geron