## 3 Proposed Approach
### Class Weights
#### 3.3.4 Transformer-Convolutional Neural Network (Transformer CNN)
This is followed by a 1D max pooling layer with a pool size of two, which reduces dimensionality, and a dropout layer with a rate of 0.000001 to mitigate overfitting. In the second hidden block, another 1D CNN layer with 512 filters and a ReLU activation function is utilized, accompanied by a 1D max pooling layer with a pool size of 4 and another dropout layer with the same rate, reinforcing regularization. The third hidden block comprises a dense layer with 1024 units and a ReLU activation function, further enhancing the model's ability  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Dataset** & **Block** & **Layer Type** & \begin{tabular}{c} **Output** \\ **Size** \\ \end{tabular}  & \begin{tabular}{c} **Activation** \\ **Function** \\ \end{tabular}  & **Parameters** & **Description** \\ \hline NF-UNSW-NB15-v2 & Input block & Input layer & (27, 1) & - & - & \begin{tabular}{c} Accepts input data with \\ 27 features. \\ \end{tabular}  \\ \hline CICIDS2017 & Input block & Input layer & (35, 1) & - & - & \begin{tabular}{c} Accepts input data with \\ 35 features. \\ \end{tabular}  \\ \hline \multirow{7}{*}{ \begin{tabular}{c} Transformer \\ block \\ \end{tabular}  } & \multirow{2}{*}{ \begin{tabular}{c} Multi-head \\ attention \\ \end{tabular}  } & \multirow{2}{*}{-} & \multirow{2}{*}{-} & \multirow{2}{*}{ \begin{tabular}{c} num\_heads = 8, \\ key\_dim = 128 \\ \end{tabular}  } & \begin{tabular}{c} Captures complex \\ relationships within \\ input data. \\ \end{tabular}  \\ \cline{1-1} \cline{3-6}  & & \begin{tabular}{c} Layer \\ Normalization \\ \end{tabular}  & - & - & epsilon = \(1\times 10^{-6}\) & \begin{tabular}{c} Normalizes the output \\ from the attention layer. \\ \end{tabular}  \\ \hline \multirow{7}{*}{ \begin{tabular}{c} S Shared Structure \\ \end{tabular}  } & \multirow{2}{*}{\ \begin{tabular}{c} Add (Residual \\ Connection) \\ \end{tabular}  } & \multirow{2}{*}{-} & \multirow{2}{*}{-} & \multirow{2}{*}{-} & \multirow{2}{*}{ \begin{tabular}{c} Adds input data to the \\ attention output for \\ stability. \\ \end{tabular}  } \\ \cline{1-1} \cline{3-6}  & & & & & & \\ \cline{1-1} \cline{3-6}  & & & & & \\ \cline  
to represent complex patterns. This block also includes a dropout layer with a rate of 0.0000001 for additional regularization. The output block varies based on the dataset. For the NF-UNSW-NB15-v2 dataset, the output layer consists of 10 units, while for the CICIDS2017 dataset, it includes 15 units. Both employ a softmax activation function to perform multi-class classification. The complete architecture, including layer specifications, is outlined in Table 30.  
(iii) Hyperparameter Configuration for the Transformer-CNN Model  
The hyperparameters for the Transformer-CNN model, detailed in Table 31, have been meticulously optimized for effectiveness in both binary and multi-class classification tasks. The model operates with a batch size of 128, which defines the number of samples processed before the model's weights are updated, ensuring consistency across both classification scenarios. The learning rate for both the binary and multi-class classifiers is dynamically adjusted using the ReduceLROnPlateau schedule. If the validation loss does not improve for a specified number of epochs (patience), the learning rate is reduced by a factor of 0.5. This strategy helps to fine-tune the model's learning process, allowing for smaller adjustments as training progresses, potentially leading to improved convergence. The learning rate is bounded below by a minimum value of \(1\times 10^{-5}\), preventing it from becoming so small that the model makes ineffective updates. This approach enhances training efficiency and stability, ensuring the model can reliably converge without overshooting the optimal solution. The Adam optimizer is utilized due to its robust adaptive learning features, demonstrating effectiveness in both binary and multi-class contexts.