## 3 Proposed Approach
### Class Weights
#### 3.3.4 Transformer-Convolutional Neural Network (Transformer CNN)
\[\textit{MultiHead}\ (Q,\ K,\ V)=\textit{Concat}\ (\textit{head}_{1}\,\ldots.\ldots,\textit{head}_{h})W^{0} \tag{23}\]  
In this context, \(W^{0}\) refers to the output weight matrix, which is utilized to transform the output of the preceding layer into the final output of the model. Layer normalization stabilizes the output of each layer, using Equation (24) [96]. \[\textit{LayerNorm}(x)=\frac{X-}{\sigma}*\gamma+\beta \tag{24}\]  
In this context, \(\mu\) represents the mean of the inputs, while \(\sigma\) denotes the standard deviation. Additionally, \(\gamma\) and \(\beta\) are learnable parameters that are utilized in the normalization process. The FFN processes the output from the attention mechanism as presented in Equation (25) [95]. \[\textit{FNN}(X)=\max(0,\ xW_{1}+b_{1})W_{2}+b_{2} \tag{25}\]  
In this context, \(W_{1}\) and \(W_{2}\) refer to the weight matrices, while \(b_{1}\) and \(b_{2}\) represent the corresponding biases associated with the layers in the model. The convolution operation in the CNN layers can be defined as in Equation (26) [89]. \[Z_{i,j}=(X*K)\ _{i,j}=\sum_{m}\sum_{n}Z_{i+m,j+n}k_{m,n} \tag{26}\]  
In this scenario, \(Z\) denotes the output feature map resulting from the convolution process, while \(X\) represents the input feature map. The convolution kernel, denoted as \(K\), is used to perform the convolution operation between the input and the output. The ReLU activation function, presented in Equation (27) [90], is efficient and straightforward, outputting zero for negative inputs while passing positive values through unchanged. Its ability to promote sparse activations and facilitate gradient flow makes it particularly effective in deep learning applications. \[\textit{ReLU}(\text{x})=\textit{max}(0,x) \tag{27}\]  
The max pooling operation can be expressed as presented in Equation (28) [89]. \[P_{i,j}=\textit{max}\big{(}X\ _{i\dot{x}+p,\ j\dot{+}q}\big{)} \tag{28}\]  
In this context, \(P\) represents the pooled output generated from the pooling operation, while \(P\) and \(q\) denote the dimensions of the pooling window applied to the input feature map. The dropout layer randomly sets a fraction \(p\) of input units to zero during training to prevent overfitting, as presented in Equation (29) [91]. \[\textit{Dropout}(x)=\left\{ \begin{aligned} & x\ \textit{with probability}\ 1-p\\ & 0\ \textit{with probability}\ \textit{p}\end{aligned} \right. \tag{29}\]  
For binary classification tasks, the sigmoid function is utilized, which outputs a probability score indicating the likelihood of an instance belonging to the positive class, as presented in Equation (30) [92]. \[\sigma\left(Z\right)=\frac{1}{1+e^{-z}} \tag{30}\]  
where \(Z\) is the output from the last dense layer. The output layer employs the softmax function for multi-class classification tasks, enabling the model to produce probability distributions across multiple classes, as presented in Equation (31) [92]. \[Softmax\;(Z_{i})=\frac{e^{z_{i}}}{\sum_{j}e^{z_{j}}} \tag{31}\]  
where \(Z_{i}\;\) is the output for class \(i\), and \(Z_{j}\;\) represents the raw score for class \(j\). (i) Binary Classification  
The architecture of the Transformer model designed for binary classification is detailed in Table 27. The model begins with an input layer that processes data structured as (25, 1) for the NF-UNSW-NB15-v2 dataset and (69, 1) for the CICIDS2017 dataset, effectively accommodating input with 25 features for NF-UNSW-NB15-v2 and 69 features for CICIDS2017. Following this, the Transformer block employs a multi-head attention mechanism with eight heads and a key dimension of 128. This mechanism captures complex relationships within the input data, enhancing the model's ability to identify intricate patterns. The output from the attention layer is subsequently normalized using layer normalization with an epsilon value of 1 \(\times\) 10\({}^{-6}\), which helps stabilize the output. A residual connection is implemented to add the original input data back to the attention output, promoting stability during training.