Top 6 Deep Learning Models You Should Master for Killer AI Applications | by Orhan G. Yalçın | Medium | Towards Data Science

The field of deep learning has gained popularity with the rise of available processing power, storage space, and big data. Instead of using traditional machine learning models, AI engineers have been gradually switching to deep learning models. Where there is abundant data, deep learning models almost always outperform traditional machine learning models. This relationship can be visualized as follows:
Therefore, as we collect more data at every passing year, it makes sense to use deep learning models. Furthermore, the field of deep learning is also growing fast. A lot of researchers put a lot of effort into developing great deep learning models for particular problems. Even though there are hundreds of alternative deep learning models, they can be grouped under major categories. These categories (referred to as models hereinafter) can be listed as follows:
Although the numbers are more than 20–30, some models are proven to be more effective for real-life problems. In this post, we will focus on these models, namely:
1 — Feedforward Neural Network Model or MLP
2 — Convolutional Neural Network Model
3 — Recurrent Neural Network Model
4 — Transformers Model
5 — Autoencoders
6 — Generative Adversarial Network Model
Let’s cover these models one by one:
FYI: The below terms are often used interchangeably:- Deep Learning Models- Artificial Neural Networks- Neural Network Architectures- Neural Network ModelsSo, don't get confused along the way :)
A feedforward neural network (the FfNN) (sometimes referred to as Regular Neural Networks or RegularNets as well) is an artificial neural network wherein the connections between the layers do not form a cycle. This is the main characteristic that makes feedforward neural networks different than recurrent neural networks. The feedforward neural network (or the multi-layer perception) was the first and is the simplest type of artificial neural network architecture.
In this network, the information moves in only the forward direction from the input layer, through the hidden layers, and to the output layer. There are no cycles or loops in an FfNN.
The feedforward neural networks we come across today are improved versions of the original & simple multilayer perceptrons (MLP). These terms are often used interchangeably. In a modern FfNN, we often use a more complex activation function than a step function (0 or 1) such as ReLU, Sigmoid, TanH, and Softmax. Modern deep neural networks usually take advantage of one of the gradient descent methods for optimization. An example of the modern feedforward neural network is shown in the Figure below:
Perceptron is a binary classification algorithm for supervised learning and consists of a layer of Linear Threshold Unit (LTU). In a Perceptron, LTUs use the same event outputs as input. The perceptron algorithm can adjust the weights to correct the behavior of the trained neural network. Besides, a bias term may be added to increase the accuracy performance of the network.
When there is only one layer of Perceptron, it is called a Single-layer Perceptron. There is one layer for outputs along with a single input layer that receives the inputs. When hidden layers are added to a single layer perceptron, we end up with a multilayer perceptron (MLP). The feedforward neural networks we build for everyday problems are complex versions of MLP.
Feedforward neural networks are excellent architectures to start learning deep learning models. Even though they have their drawbacks, feedforward neural networks can be used to build speech-recognition, image-recognition, and machine-translation software. But, they are most powerful in data analytics problems. Here is another visualization of feedforward neural networks:
Here is a Feedforward Neural Network case study that you can use to polish your FfNN skills:
towardsdatascience.com
One of the most powerful supervised deep learning models is the Convolutional Neural Networks (the CNNs). The final structure of a CNN is actually very similar to Feedforward neural networks (FfNNs), where there are neurons with weights and biases. Besides, just like in FfNNs, there is a loss function (e.g., crossentropy or softmax) and an optimizer (e.g., adam optimizer) in CNNs. Additionally, though, in CNNs, there are also Convolutional Layers, Pooling Layers, and Flatten Layers. CNNs are mainly used for computer vision tasks such as image classification, although you may find other application areas such as natural language processing and time series prediction.
We are capable of using many different layers in a convolutional neural network. However, convolution, pooling, and fully connected layers are the most important ones. Therefore, I will quickly introduce these layers before implementing them.
Convolutional Layer — The convolutional layer is the first layer where we extract features from our datasets' images. Since pixels are only related to the adjacent and close pixels, convolution allows us to preserve the relationship between different parts of an image. Convolution is basically filtering the image with a smaller pixel filter to decrease the image's size without losing the relationship between pixels. When we apply convolution to a 5x5 image using a 3x3 filter with 1x1 stride (1-pixel shift at each step), we will have a 3x3 output (64% decrease in complexity).
Pooling Layer — When constructing CNNs, it is common to insert pooling layers after each convolution layer to reduce the representation's spatial size to reduce the parameter counts, which reduces the computational complexity. Besides, pooling layers also helps with the overfitting problem. Basically, we select a pooling size to reduce the amount of the parameters by selecting the maximum, average, or sum values inside these pixels. Max Pooling, one of the most common pooling techniques, may be demonstrated as follows:
A Set of Fully Connected Layers — A fully connected network is our FfNN where each parameter is linked to one another to determine each parameter's true relation and effect on the labels.
The final network with convolution layers, pooling layers, and a set of fully-connected layers looks like this:
Here is a Convolutional Neural Network case study that you can use to polish your CNN skills:
towardsdatascience.com
A recurrent neural network (RNN) is a class of artificial neural networks where connections between neurons form a temporal sequence. RNNs capable of capturing dynamic temporal information (temporal memory). Recurrent Neural Networks are derived from Feedforward Neural Networks, but they offer much more.
RNNs use previous information by keeping them in memory, which is saved as a “state” within an RNN neuron. There are several types of RNNs, such as Vanilla RNN, GRU, and LSTM.
There are many different variants of RNNs. The three RNN types we encounter often are
Vanilla (Simple) RNN;
Long Short-term Memory (LSTM) Networks; and
Gated Recurrent Unit (GRU) Networks.
You can find the visualization of these alternative RNN cells in Figure 7:
Especially LSTM and GRU nodes are so widely used, you often see them as separate deep learning model types in some blog posts.
Here is a Recurrent Neural Network case study that you can use to polish your CNN skills:
towardsdatascience.com
The Transformer networks are deep learning models that are introduced in 2017. Transformer networks are primarily used for natural language processing (NLP) tasks.
Transformers were designed as a response to the computational complexity and slowness of RNNs. Transformers can handle sequential data tasks such as machine translation, time series prediction, or text summarization.
Transformers' main advantage is that, unlike RNNs, they don’t require sequential data to be processed in sequence. If the input data is sales figures in a time-series, the Transformer does not need to process the earlier dates before the later dates. Therefore, the Transformers allow for much more parallelization than RNNs and therefore reduced training times.
Since they are much more efficient in terms of training speed, Transformers have become NLP researchers' choice and replace RNN alternatives. For example, popular pre-trained NLP models such as BERT and GPT-3 are transformer models that you can use for your NLP tasks with transfer learning.
Here is a Transformers Network case study that you can use to polish your Transformers skills:
towardsdatascience.com
Autoencoders are neural network architectures that consist of two sub-networks, namely, encoder and decoder networks, which are tied to each other with a latent space. Autoencoders were first developed by Geoffrey Hinton, one of the most respected scientists in the AI community, and the PDP group in the 1980s. Hinton and the PDP Group aimed to address the “backpropagation without a teacher” problem, a.k.a. unsupervised learning, using the input as the teacher. In other words, they used feature data both as feature data and label data.
Let’s take a closer look at how autoencoders work!
Autoencoders consists of an encoder network, which takes the feature data and encodes it to fit into the latent space and a decoder network. The decoder uses this encoded data (i.e., code) and converts it back to the feature data. In an encoder, the model learns how to encode the data efficiently so that the decoder can convert it back to the original. Therefore, the essential part of autoencoder training is to generate an optimized latent space.
Now, know that in most cases, the number of neurons in the latent space is much smaller than the input and output layers, but it does not have to be that way. There are different types of autoencoders, such as undercomplete, overcomplete, sparse, denoising, contractive, and variational autoencoders.
When building an autoencoder, the standard practice is to design an encoder and create an inversed version of this network as the decoder of this autoencoder. So, as long as there is an inverse relationship between the encoder and the decoder network, you are free to add any layer to these sub-networks. For example, if you are dealing with image data, you would surely need convolution and pooling layers. On the other hand, if you are dealing with sequence data, you would probably need LSTM, GRU, or RNN units. The important point here is that you are free to build anything you want. Here is a common way of building an autoencoder network:
Here is an Autoencoder Network case study that you can use to polish your Autoencoder skills:
towardsdatascience.com
Generative Adversarial Networks offer revolutionary generative solutions that can be used in various areas such as protein modeling to art generation. The invention of GANs happens pretty unexpectedly. The famous AI researcher, then, a Ph.D. fellow at the University of Montreal, Ian Goodfellow, lands on the idea when he was discussing with his friends -at a friend’s going away party- about the flaws of the other generative algorithms. After the party, he goes home with high hopes and implements the concept he had in mind. Surprisingly, everything goes as he hoped in the first trial, and he successfully creates the Generative Adversarial Networks (shortly, GANs). According to Yann Lecun, the director of AI research at Facebook and a professor at New York University, GANs are “the most interesting idea in the last 10 years in machine learning” [6].
The rough structure of the GANs may be demonstrated as follows:
There are two agents competing with each other in an ordinary GAN structure: a Generator and a Discriminator. They may be designed using different networks (e.g., Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), or just Feedforward Neural Networks (FfNN or RegularNets)). A CNN based GAN can be visualized as follows:
As an example, for an image generation task, we ask the generator to generate images without giving it any additional data. Simultaneously, we fetch real images from our dataset to the discriminator and ask it to decide whether the images generated by the Generator are genuine or not. At first, the Generator outputs noises that will immediately be labeled as fake by the Discriminator. The Generator eventually learns to trick the Discriminator after getting enough feedback from the Discriminator. As a result, we obtain an excellent generative model that can give us very realistic outputs.
Here is a Generative Adversarial Network case study that you can use to polish your GAN skills:
towardsdatascience.com
This post covered the top 6 popular deep learning models that you can use to build great AI applications. By using the respective case studies, you can build AI applications for:
Predictive Analytics using an FfNN;
Image Classification using a CNN;
Time-series Price Prediction using an RNN;
Sentiment Analysis using Transformers;
Noise Reduction using Autoencoders; and
Image Generation using a GAN.
So, get to work and polish your skills on your way to become a great deep learning expert!
If you would like to have access to the codes of my other tutorial posts on Google Colab, and have early access to my latest content, consider subscribing to the mailing list:✉️
Subsribe Now
If you are interested in deep learning, also check out the guide to my content on artificial intelligence:
oyalcin.medium.com
Since you are reading this article, I am sure that we share similar interests and are/will be in similar industries. So let’s connect via Linkedin! Please do not hesitate to send a contact request! Orhan G. Yalçın — Linkedin