## 3 Proposed Approach
### Class Weights
#### 3.3.4 Transformer-Convolutional Neural Network (Transformer CNN)
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & **Block** & **Layer Type** & **Output Size** & **Activation** & **Parameters** & **Description** \\ \hline \multirow{4}{*}{Shared Structure} & Transformer block & Multi-head attention & - & - & num\_heads = 8, key\_dim = 128 & Captures complex relationships within input data. \\ \cline{2-5}  & Layer Normalization & - & - & epsilon = \(1\times 10^{-6}\) & Normalizes the output from the attention layer. \\ \hline \multirow{4}{*}{Shared Structure} & Add (Residual Connection) & - & - & - & Adds input data to the attention output for stability. \\ \cline{2-5}  & Feed Forward block & Dense layer & 512 & ReLU & \begin{tabular}{c} units = 512, activation = ’relu’ \\ \end{tabular} & \begin{tabular}{c} Applies a dense transformation with ReLU activation. \\ \end{tabular} \\ \cline{2-5}  & Dropout layer & - & - & rate = 0.0000001 & \begin{tabular}{c} Regularizes the network to prevent overfitting \\ (\(p\) = 0.00000001). \\ \end{tabular} \\ \cline{2-5}  & Dense layer & 512 & - & units = 512 & \begin{tabular}{c} Another dense transformation without activation. \\ \end{tabular} \\ \hline \multirow{4}{*}{Shared Structure} & Dropout layer & - & - & rate = 0.0000001 & \begin{tabular}{c} Further regularization \\ (\(p\) = 0.0000001). \\ \end{tabular} \\ \cline{2-5}  & Add (Residual Connection) & - & - & - & \
\begin{tabular}{c} Adds feed-forward \\ output to the previous block output. \\ \end{tabular} \\ \hline \multirow{4}{*}{Shared Structure} & Layer Normalization & - & - & epsilon = \(1\times 10^{-6}\) &
\begin{tabular}{c} Normalizes the combined output \\ for stability. \\ \end{tabular} \\ \cline{2-5}  & & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 27: _Cont._ \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & **Block** & **Layers** & **Layer Size** & **Activation** \\ \hline \multirow{8}{*}{Shared Structure} & Input block & Input layer & Transformer output & - \\ \cline{2-5}  & Hidden block 1 & 1D CNN layer & 512 & ReLU \\ \cline{2-5}  & & 1D Max Pooling layer & 2 & - \\ \cline{2-5}  & & Dropout layer & 0.0000001 & - \\ \cline{2-5}  & Hidden block 2 & 1D CNN layer & 512 & ReLU \\ \cline{2-5}  & & 1D Max Pooling layer & 4 & - \\ \cline{2-5}  & Dropout layer & 0.0000001 & - \\ \cline{2-5}  & Hidden block 3 & Dense layer & 1024 & ReLU \\ \cline{2-5}  & Dropout layer & 0.0000001 & - \\ \cline{2-5}  & Output block & Output layer & 1 & Sigmoid \\ \hline \hline \end{tabular}
\end{table} 
Table 28: CNN model layers for binary classification. from the attention layer is then normalized using layer normalization with an epsilon value of \(1\times 10^{-6}\), which contributes to stabilizing the output. A residual connection is established to add the original input data back to the attention output, promoting stability during training. The feed-forward block consists of a dense layer with 512 units and a ReLU activation function, which applies a transformation to the data. This is followed by a dropout layer with a rate of 0.0000001, designed to mitigate overfitting by regularizing the network. An additional dense layer with 512 units is included without an activation function, allowing for further transformations. A subsequent dropout layer with the same rate reinforces regularization, enhancing the model's robustness. The output from the feed-forward network is then added back to the previous block's output via another residual connection, followed by an additional layer normalization step with epsilon = 1 \(\times\) 10\({}^{-6}\) to normalize the combined output, ensuring stability in the model's learning process. The architecture of the CNN model designed for multi-class classification leverages the output of the Transformer model as its input, tailored for both the NF-UNSW-NB15-v2 and CICIDS2017 datasets. The model starts with an input block that processes the Transformer output. The first hidden block incorporates a 1D CNN layer with 512 filters and a ReLU activation function, enabling the extraction of critical features from the data.