## 3 ENN
### Architectures of Models
#### 3.3.5 Transformer-Deep Neural Network (Transformer-DNN)  
The model architecture integrates transformer and DNN components to enhance classification performance by processing sequential data, where each data point is represented as a 1D vector. The transformer block plays a crucial role in capturing intricate relationships within the input data, using its multi-head attention mechanism, dynamically assigning varying levels of importance to different parts of the input sequence and aiding in the identification of complex patterns and dependencies. After the attention mechanism, the output is passed through a feed-forward network (FFN) consisting of dense layers with non-linear transformations, thereby refining the feature representations further. Residual connections maintain gradient flow during training, improving convergence, while layer normalization stabilizes the output. After the transformer block, the output is passed to DNN layers, where multiple fully connected layers with ReLU activation progressively refine the feature representations learned by the transformer, enabling the model to learn more complex and abstract patterns. Dropout layers are applied to regularize the network and prevent overfitting by randomly disabling a proportion of neurons during training. \begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Parameter** & **Binary Classifier** & **Multi-Class Classifier** \\ \hline Batch size & 128 & 128 \\ \hline \multirow{2}{*}{Learning rate} & Scheduled: Initial = 0.001, & Scheduled: Initial = 0.001, \\  & Factor = 0.5, Min = \(1\times 10^{-5}\) & Factor = 0.5, Min = \(1\times 10^{-5}\) \\  & (ReduceLROnPlateau) & (ReduceLROnPlateau) \\ \hline Optimizer & Adam & Adam \\ \hline Loss function & Binary cross-entropy & Categorical cross-entropy \\ \hline Metric & Accuracy & Accuracy \\ \hline \hline \end{tabular}
\end{table} 
Table 24: Hyperparameters of the CNN model. \begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Parameter** & **Binary Classifier** & **Multi-Class Classifier** \\ \hline Batch size & 64 & 64 \\ \hline Learning rate & 0.001 & 0.001 \\ \hline Optimizer & Adam & Adam \\ \hline Loss function & Mean\_squared\_error & Mean\_squared\_error \\ \hline Metric & Accuracy & Accuracy \\ \hline \hline \end{tabular}
\end{table} 
Table 23: Hyperparameters of the Autoencoder model. The output is then flattened and further processed by the DNN. The final output layer uses a sigmoid activation function for binary classification, producing a probability score between 0 and 1, indicating the likelihood of an instance belonging to the positive class. For multi-class classification, the output layer employs a softmax activation function with a number of neurons equal to the number of classes, generating a probability distribution over all classes and ensuring the sum of the probabilities equals 1. This distribution reflects the likelihood of an instance belonging to each class, with the highest probability indicating the predicted class. The combination of the global context and long-range dependencies captured by the transformer along with the high-level feature extraction and non-linear transformations provided by the DNN results in an effective architecture for both binary and multi-class classification tasks, enabling the model to leverage both global and local patterns within the data for classification into either two or multiple categories. The multi-head attention mechanism efficiently identifies intricate relationships within the input data, as demonstrated by Equation (32) [56]. \[Attention(Q,\ K,\ V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}}) \tag{32}\]  
where Q represents the query matrix, K denotes the key matrix, and V signifies the value matrix. The variable \(d_{k}\) refers to the dimension of the keys, which plays a crucial role in the computation of attention scores within the model. Each attention head computes its attention independently, and the results are then concatenated according to Equation (33) [56].