An Overview of Deep Learning — from History to Fundamentals | by Pedram Ataee, PhD | Towards Data Science

I recently taught a mini-course on Machine Learning 101 for those who want to become data scientists. One of its modules was about Deep Learning. I found that many newbies are confused with this topic mostly because it is often taught with many complexities. In this article, I aim to describe it simple enough but not too simple. Hope it helps!
The article has 4 sections as follow:
What is Neural Network?
What is Deep Learning?
How to build a simple deep learning architecture?
How to train a deep learning model?
A neural network is a computational model inspired by the biological neural network that processes information in the human brain. A neural network consists of a set of artificial neurons organized in layers (input, hidden, and output). These artificial neurons are connected by synapses that are just weighted values.
An artificial neuron is a computing element with a specific mathematical operation. A neuron takes its input (boolean or real) and, after applying the pre-defined operation on the input, passes the result through an activation function to the other neurons. The activation function is the part of each node that turns a linear function into a non-linear function. Basically, it decides whether a neuron should fire or not. The activation function can be different mathematical functions such as Step, Sigmoid, and ReLU.
A common neuron (a.k.a., a perceptron neuron) takes inputs, including a bias, and multiplies them by associated weights, and then passes the sum through the step function.
Training a neural network refers to a process that calculates (or, finds) the weights in the network to minimize the objective function (or, loss function). You can think of weights in the neural network as parameters in a polynomial function such as ax^2+b*x+c with two main differences. The neural network is a polynomial function with a high degree and potentially nonlinear performance. Here, a high degree means the number of parameters can exceed millions when needed.
There are various algorithms to train a neural network, and backward propagation (backpropagation) is one of them. In short, backpropagation is an algorithm that uses the gradient descent technique to calculate weights in a neural network during the training stage. In this method, the errors of outputs in comparison to the desired values are computed and proportionally (i.e., based on the current weights) propagated in a backward direction to update the network weights. The network weights are iteratively updated based on a learning rate until convergence.
I recommend watching the video to learn more about this method. The 3Blue1Brown YouTube channel has one of the best visualizations to describe this topic.
To train a neural network, you must know the answers to the questions below. You may not understand their importance until start implementing a sample project.
How to initialize weights?
What are the stopping criteria?
What if the algorithm doesn’t converge?
Does adding complexity to the network help?
One last thing. If you want to find out how the configuration parameters such as learning rate, activation function, or architecture affect the outcome of a neural network, I highly recommend checking the interactive demo named Playground created by TensorFlow. It is so beautiful and insightful!
playground.tensorflow.org
Deep learning is a series of machine learning methods based on special architectures of deep neural networks (neural networks with many hidden layers) that can conduct both feature extraction and classification in unison and with little human effort. The special architectures are more advanced than layers in neural networks that are simply fully connected. The special architectures are mostly built upon a concept named “capsule”. A capsule is a group of neurons in each layer that do a lot of internal computation and output a compact result representing a property of data such as Convolution. You can read more about the Convolutional Neural Network
AlphaGo project created by Deepmind is one of the success stories of deep learning. As stated by Deepmind: “AlphaGo is the first computer program to defeat a Go world champion”. First, let me describe why Go is special.
Go is the most challenging classic game to be targeted by artificial intelligence. Why? Mostly due to the number of moves a player can choose. According to this article, after the first 2 moves, there are about 130,000 moves in Go. This number is at 400 possible moves in chess. You can see that the search space in Go is unarguably vast. Deep learning can be a good choice when the search space is drastically vast.
Another reason why deep learning is the right method to target the Go is rooted in how Go is being played. If you ask Go players how they decide on a move, they often tell you it just felt right. In these sorts of scenarios when you can not define features, you can not use classical machine learning algorithms. Deep learning can be a good choice when is not feasible to determine a useful feature set.
A convolutional neural network (or, CNN) is a class of deep learning architectures commonly used to analyze images such as image classification, object detection, and video action recognition. In general, the convolutional neural network is designed to be used with any data that has some spatial invariance in its structure such as face or speech recognition. Spatial-invariance means that, for example, a cat ear in the top left of the image has the same features like a cat ear in the bottom right of the image. CNNs are built upon two main building blocks described below.
Convolution — CNNs are spatially invariant since they are built upon the convolution operator. Convolution is a mathematical operation that does the integral of the product of two functions (signals), with one of the signals flipped (if needed). In the past, convolution operators have been used, for example, to compute the correlation between two signals or find patterns in signals. This operator functions well for feature extraction in visual data. Much of the progress made in computer vision over the past few years is due in part to convolutional neural networks.
Pooling — Another building block in CNNs is a pooling layer. Its function is to progressively reduce the spatial size of the data to reduce the network size and the algorithm sensitivity to the precise position of the feature in the input. The network size translates to the number of weights that must be calculated during the training phase.
You sometimes also need to add extra pixels with pixel_intensity=0 around the boundary of the input images to increase the effective size. That helps to keep image size fixed after applying convolutional layers. This process is called padding.
AlexNet —AlexNet is a successful implementation of convolutional neural networks that won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2012. The architecture has been published in NeurIPS 2012 by Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton (Figure 2). In this architecture, the input is an RGB image of size 256×256 which were randomly cropped into size 224×224. The architecture contains 650,000 neurons and 60 million parameters. Plus, it took 5-6 days to train on two GTX 580 3GB GPUs. It consists of 5 Convolutional Layers and 3 Fully Connected Layers. They used Rectified Linear Units (ReLUs) as activation functions for the first time.
VGG16 — A standard way to improve the performance of deep neural networks is by increasing the depth. VGG-16 was invented by folks at Visual Geometry Group at the University of Oxford. This architecture has 13 convolutional and 3 fully-connected layers. They also used the ReLU activation function as the tradition carrying over from AlexNet. This network stacks more layers compared to AlexNet and uses smaller size filters (2×2 and 3×3). It consists of 138M parameters.
towardsdatascience.com
ResNet50 — With the network depth increasing, accuracy gets saturated and then degrades rapidly mostly due to the fact that we can not train it properly. Microsoft Research addressed this problem with ResNet50 — using skip (or, shortcut) connections, while building deeper models. A deeper CNNs (up to 152 layers) without compromising model generalization. This is a great idea ....
Keras is a high-level neural-networks library with a Python interface that can be run on top of major scientific computing frameworks such as TensorFlow (created by Google) or CNTK (created by Microsoft). Engineers often prefer Keras providing fast development compared to Pytorch. You can find how to build a VGG-like convolutional neural network using Kears below. To learn more about how to build a deep learning model using Keras, you can check out the original documentation.
from keras.models import Sequentialfrom keras.layers import Dense, Dropout, Flattenfrom keras.layers import Conv2D, MaxPooling2Dfrom keras.optimizers import SGDmodel = Sequential()model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(100, 100, 3)))model.add(Conv2D(32, (3, 3), activation='relu'))model.add(MaxPooling2D(pool_size=(2, 2)))model.add(Dropout(0.25))model.add(Conv2D(64, (3, 3), activation='relu'))model.add(Conv2D(64, (3, 3), activation='relu'))model.add(MaxPooling2D(pool_size=(2, 2)))model.add(Dropout(0.25))model.add(Flatten())model.add(Dense(256, activation='relu'))model.add(Dropout(0.5))model.add(Dense(10, activation='softmax'))sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)model.compile(loss='categorical_crossentropy', optimizer=sgd)model.fit(x_train, y_train, batch_size=32, epochs=10)
PyTorch is a python-based, low-level neural-networks library built upon Facebook scientific computing framework (Torch). Its workflow is similar to Python scientific computing library (Numpy). Pytorch is highly configurable and is more popular among researchers compared to developers who don’t need complex architecture or special layer operations. You can find how to build a convolutional neural network using PyTorch below. To learn more about how to build a deep learning model using PyTorch, you can check out the original documentation.
import torch.nn as nnimport torch.nn.functional as Fclass Net(nn.Module):    def __init__(self):        super().__init__()        self.conv1 = nn.Conv2d(3, 6, 5)        self.pool = nn.MaxPool2d(2, 2)        self.conv2 = nn.Conv2d(6, 16, 5)        self.fc1 = nn.Linear(16 * 5 * 5, 120)        self.fc2 = nn.Linear(120, 84)        self.fc3 = nn.Linear(84, 10)    def forward(self, x):        x = self.pool(F.relu(self.conv1(x)))        x = self.pool(F.relu(self.conv2(x)))        x = torch.flatten(x, 1)         x = F.relu(self.fc1(x))        x = F.relu(self.fc2(x))        x = self.fc3(x)        return xnet = Net()
It is impossible to explain everything about training a neural network in this article. Here, I want to shed light on the most important topics.
As described above, training a neural network refers to a process that calculates the weights in the network to minimize the objective function. This is a classical optimization problem where you must search for the optimum set of weights (or, parameters) that minimize the loss function. The efficacy of the search method determines the speed and outcome of the training process.
Food For Thought — In machine learning algorithms, we choose a metric (e.g., accuracy) to evaluate a model; however, we optimize a different objective function and “hope” that minimizing its value will improve metric we care about. So, what can we do to ensure reaching to the desired requirements?
In general, to implement a search method, you must have answers to the following questions: (1) “How to determine the search direction?”, and (2) “How to determine the search step?”.
As described above, the gradient descent technique has been used in the training of neural networks to guide the search process. The search direction is determined by the gradient operator and the search step is determined by a hyperparameter λ that is also called the learning rate. In short, the update mechanism in the gradient descent technique is as follows: xm=xn -λ*∇f(xn). The classic gradient descent techniques can not be simply used for deep learning techniques where a much higher number of weights (parameters) and a number of data points often exist. ⛔
🚀Stochastic gradient descent (SGD) is a variation of the gradient descent technique that is more efficient for deep learning models. This method calculates the error and updates the model for each data point in the training dataset in contrary to computing error using batch data. The SGD is fast in search and offers more frequent model updates. On the other hand, the SGD results are different in each run and it has slower convergence in comparison to the standard technique.
Instead of using only the gradient of the current step to guide the search, we can use the gradient of the past steps knowing that the most recent ones are more important. This is called Momentum in the optimization context. Therefore, we can, for example, use an exponential average of the gradient steps to have a more efficient search process. Check out the article below to learn more about Momentum.
distill.pub
We have other techniques to solve the optimization problem such as AdaGrad or Adam (Adaptive Moment Estimation). These methods are variations of gradient descent optimization that adaptively change the learning rate to ensure having a more efficient search process. In simple words, the learning rate or search step can be different for each direction in the search space as well as at each moment in time. You can read more here. If you want to learn how to use these techniques in the real world, you can take a look at the Keras official documentation.
If you like this post and want to support me...
Follow me on Medium!
Check out my books on Amazon!
Become a member on Medium!
Connect on Linkedin!
Follow me on Twitter!