## 5 Machine Learning Model Designs
### Regression and Classification Problems
#### 5.1.7 Neural Networks
There are a few different formulations for simple RNN networks, such as the following: The \(W\), \(I\), and \(b\) are the weights and recurrent connection matrices. \[ \begin{split}& h_{t}=\sigma_{h}(W_{h}i_{t}+X_{h}h_{t-1}+b_{h})\\ & O_{t}=\sigma_{O}(W_{o}h_{t}+b_{o})\end{split} \] The hidden layer vector is calculated, including the recurrent connections as \(h_{t}\). The output vector is \(O_{t}\), and \(i_{t}\) is the input vector. The activation functions are \(\sigma_{h}\) and \(\sigma_{O}\)[34]. 3. Long Short-Term Memory (LSTM): Long short-term memory is an artificial neural network cell type that maintains memory and has input-output and forgets gates as shown in Figure 14. The LSTM is a further developed form of the RNN that incorporates the memory cell. The memory is able to carry forward over various temporal intervals and learn relations between dependent events that are not directly temporally sequential. The LSTM is also an enhanced ANN cell for overcoming the vanishing gradient problem in deep ANNs. The vanishing gradient problem refers to the exponentially smaller error signal that propagates back through the ANN during the supervised learning process as small error values (0,1) are multiplied together. Through its cell formulation, the forget gate does not have an exponential decay factor and thus counters the vanishing gradient problem. ANN using LSTM has shown record-breaking performance in language processing, such as speech recognition, in chatbots for responding to input text, and machine translation. The formulation of an LSTM cell is as follows: let \(W\) represent weight matrices and \(U\) represent the cell's connection matrices. The components of the cell are the forget gate, input gate, output gate to the next recurrent cell in the layer, and memory cell input gate, represented respectively in the equations below. \[ \begin{array}{l}f_{t}=\sigma_{f}(W_{t}x_{t}+U_{t}h_{t-1}+b_{f})\\ i_{t}=\sigma_{i}(W_{i}x_{t}+U_{t}h_{t-1}+b_{i})\\ o_{t}=\sigma_{o}(W_{o}x_{t}+U_{o}h_{t-1}+b_{o})\\ v_{t}=\sigma_{v}(W_{v}x_{t}+U_{t}h_{t-1}+b_{v})\end{array} \]  
The cell state is maintained as \(c_{t}=f_{t}\circ c_{t-1}+i_{t}\circ v_{t}\). The output vector that goes to the next layer in the network \(h_{t}=o_{t}\circ\sigma_{h}(c_{t})\). The variable state of the memory cell is an element of real numbers, as with the network input \(c_{t}\in\mathbb{R}\) and \(x_{t}\in\mathbb{R}\). All vectors except the hidden state, output, and cell input activation vectors range from (0,1). The hidden state \(h_{t}\) is in the range of \((-1,1)\)[35; 36; 37]. 4. Autoencoders: Autoencoders are an unsupervised learning technique in which neural networks are utilized for representation learning. There is a neural network architecture that imposes a bottleneck in the network and forces a compressed knowledge representation of the original input as shown in Figure 15. If the input features were independent of one another, this compression and subsequent reconstruction would be very difficult. However, if a structure exists in the data, such as correlations between input features, this structure can be learned and applied when moving the input through the network's bottleneck. An auto-encoder comprises two parts that are encoders and decoders [38]. The objective of the encoder is to compress the data in such a way that only minimal information is lost, whereas the objective of the decoder is to utilize the compressed information and regenerate the original data. Data compression, image denoising, dimensionality reduction, feature extraction, new data generation, image colorization, and anomaly detection are a few applications of autoencoders. The basic autoencoder consists of five parts: input layer, encoder hidden layer, compressed data, also known as a latent vector, decode hidden layer, and output layer.