## 3 Proposed Approach
### Class Weights
#### 3.3.4 Transformer-Convolutional Neural Network (Transformer CNN)
The feed-forward block consists of a dense layer with 512 units and a ReLU activation function, applying a transformation to the data. This is followed by a dropout layer with a rate of 0.0000001, aimed at mitigating overfitting by regularizing the network. Another dense layer with 512 units is included without an activation function, allowing for additional transformations. A subsequent dropout layer with the same rate further reinforces regularization, enhancing model robustness. The output from the feed-forward network is then added back to the previous block's output via another residual connection, followed by another layer normalization step with epsilon = 1 \(\times\) 10\({}^{-6}\) to normalize the combined output, ensuring stability in the model's learning process. The architecture of the CNN model designed for binary classification utilizes the output of the Transformer model as its input and is tailored for datasets like NF-UNSW-NB15-v2 and CICIDS2017. The input block processes the Transformer output, providing structured input for the model. The first hidden block includes a 1D CNN layer with 512 filters and a ReLU activation function, which extracts essential features from the input data. This is followed by a 1D max pooling layer with a pool size of two, reducing the dimensionality of the feature maps, and a dropout layer with a rate of 0.0000001 to mitigate overfitting. The second hidden block repeats this structure with another 1D CNN layer with 512 filters and ReLU activation, followed by a 1D max pooling layer with a pool size of four and a dropout layer with the same dropout rate. In the third hidden block, the model incorporates a dense layer with 1024 units and a ReLU activation function, enhancing the model's representational capabilities. A dropout layer with a rate of 0.0000001 is again applied for additional regularization. The architecture concludes with a single-output layer employing a sigmoid activation function for binary classification, producing a probability score to determine class membership. The detailed structure, including layer sizes, activation functions, and dropout rates, is outlined in Table 28. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Dataset** & **Block** & **Layer Type** & **Output Size** & \begin{tabular}{c} **Activation** \\ **Function** \\ \end{tabular} & **Parameters** & **Description** \\ \hline NF-UNSW-NB15-v2 & Input block & Input layer & (25, 1) & - & - & \
\begin{tabular}{c} Accepts input data with \\ 25 features. \\ \end{tabular} \\ \hline CICIDS2017 & Input block & Input layer & (69, 1) & - & - &
\begin{tabular}{c} Accepts input data with \\ 69 features. \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table} 
Table 27: Transformer model layers for binary classification. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & **Block** & **Layers** & **Layer Size** & **Activation** \\ \hline \multirow{8}{*}{Shared Structure} & Input block & Input layer & Transformer output & - \\ \cline{2-5}  & Hidden block 1 & 1D CNN layer & 512 & ReLU \\ \cline{2-5}  & & 1D Max Pooling layer & 2 & - \\ \cline{2-5}  & & Dropout layer & 0.0000001 & - \\ \cline{2-5}  & Hidden block 2 & 1D CNN layer & 512 & ReLU \\ \cline{2-5}  & & 1D Max Pooling layer & 4 & - \\ \cline{2-5}  & & Dropout layer & 0.0000001 & - \\ \cline{2-5}  & Hidden block 3 & Dense layer & 1024 & ReLU \\ \cline{2-5}  & & Dropout layer & 0.0000001 & - \\ \cline{2-5}  & Output block & Output layer & 1 & Sigmoid \\ \hline \hline \end{tabular}
\end{table} 
Table 28: CNN model layers for binary classification.