## 5 Machine Learning Model Designs
### Regression and Classification Problems
#### 5.1.7 Neural Networks
In autoencoders, the number of input features is always equal to the number of output features, and the output layer will always have fewer neurons than the input layer; otherwise, there is the possibility of data replication and there won't be any compression. Mathematically, autoencoders can be expressed as shown in Equation (23)  
\[x\rightarrow\underbrace{f(x)\to z}_{\text{encoding}}\rightarrow \underbrace{g(z)\rightarrow\bar{x}}_{\text{decoding}}. \tag{23}\]  
\(x\) is the input vector, \(f(x)\) is the encoder function, \(z\) is the compressed data, \(g(z)\) is the decoder function and \(\bar{x}\) is the output. Deep autoencoders are similar to basic autoencoders, but they have multiple hidden layers for encoders and decoders, where the number of hidden layers for both the encoder and the decoder should be equal and should have the same number of neurons. It's similar to a mirror image, where the latent variable vector acts as a mirror. Some common types of autoencoders include denoising the autoencoder, the variational autoencoder, and the convolutional autoencoder [6]. 5. Generative Adversarial Network (GAN): Like autoencoders, the primary application of GAN is to generate a new dataset from a given set of data. However, the architecture of GAN is very different from autoencoders. In GAN, there are two parts: the generator and discriminator. These are two different neural networks that run in parallel. The generator starts with random noise and generates a sample. Based on the feedback from the discriminator, the generator trains to generate more realistic data, and the discriminator learns from the real sample of data and the feedback from the generator. With the improvement in the performance of the generator, the performance of the discriminator deteriorates, and for a perfect generator, the accuracy of the discriminator is 50%. The basic architecture is shown in Figure 16. In GAN, two loss functions work in parallel, one for the generator and the other for the discriminator. The generator's objective is to minimize the loss, whereas the objective of the discriminator is to maximize the loss. The mathematical formulation for the loss function is shown in Equation (24). \[min^{gen}max^{dis}E_{x}[\log(D(x))]+E_{z}[\log(1-D(G(z)))] \tag{24}\]  
where \(D(x)\) is the discriminators' probability that the real data is real, \(E_{x}\) is the expected value of the real data, \(z\) is the noise, \(G(z)\) is the generator output with \(z\), \(D(G(z))\) is discriminator's probability estimate that the fake data is real, \(E_{z}\) is the expected output of the random samples from the generator. 6. Convolutional networks (CNN): CNNs have great applications in computer vision. These networks receive an image as input and perform a sequence of convolution and max pool operations to progressively smaller scales, descending the initial part of the 'U' followed by up-convolution and convolution in sequence to ascend the later part of the 'U'. This model design has proven highly accurate at classifying objects in an image. The convolution network captures spatial and temporal dependencies in an image and performs better at classifying image data when compared to a flattened feed-forward neural network. In a convolutional network, a kernel or matrix passes over the 2-D data array and performs an up or down convolution on the data; alternatively, in a color image, the convolution kernel can be 3-dimensional. These networks are trained in supervised learning and classify images using the softmax classification technique. The ability to automatically do feature extraction and determination is the strength of these models over any competing model. The top image classification AI is based on CNNs, including LeNet, AlexNet, VGGNet, GoogleLeNet, ResNet, and ZFNet. 7. Transformer Neural Network: Transformer networks are more effective at handling long data sequences because they handle the incoming data in parallel, while standard neural networks process the data sequentially.