## 3 Proposed Approach
### Class Weights
#### 3.3.4 Transformer-Convolutional Neural Network (Transformer CNN)  
The model architecture presented integrates both Transformer and CNN components to enhance classification performance for the given task. The input layer receives data in a structured format, setting the stage for the subsequent processing. The Transformer block plays a crucial role in capturing intricate relationships within the input data through its multi-head attention mechanism. This approach allows the model to weigh different parts of the input more dynamically, facilitating the identification of complex patterns and dependencies. To stabilize the learning process and improve gradient flow, a layer normalization and residual connection are employed. Following the attention mechanism, a feed-forward neural network (FFN) processes the output, enhancing the data representation by applying non-linear transformations and introducing dropout layers for regularization. The primary job of the Transformer is to provide a global context and highlight important features across the entire input sequence. Subsequently, the CNN blocks operate on the output of the Transformer, focusing on local feature extraction. Each convolutional layer applies filters to detect various features within the input data, while batch normalization and activation functions such as ReLU ensure that the model remains robust and learns effectively. Max pooling layers downsample the data, reducing its dimensionality and allowing the model to concentrate on the most salient features. The CNN's primary function is to capture spatial hierarchies and patterns within the input, making it particularly effective for tasks requiring detailed analysis of local structures. The architecture also includes a flattening step that prepares the output of the CNN blocks for further processing. This flattened representation is then passed through MLP blocks, which serve to learn high-level abstractions from the features extracted by the CNNs. The concatenation of the Transformer and CNN outputs at this stage enables the model to leverage both global context and local feature patterns for improved classification accuracy. Finally, the output layer employs a sigmoid or softmax activation function to generate class probabilities, completing the model's capability to classify inputs based on the rich representations learned throughout the architecture. This integrated approach harnesses the strengths of both Transformer and CNN architectures, providing a comprehensive framework for effective classification tasks. The multi-head attention mechanism effectively captures complex relationships within the input data, as represented by Equation (22) [95]. \[Attention\ (Q,\ K,\ V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right) \tag{22}\]  
In this context, \(Q\) represents the query matrix, \(K\) denotes the key matrix, and \(V\) signifies the value matrix. The variable \(d_{k}\) refers to the dimension of the keys, which plays a crucial role in the computation of attention scores within the model. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Parameter** & **Binary Classifier** & **Multi-Class Classifier** \\ \hline Batch size & 128 & 128 \\ \hline \multirow{3}{*}{Learning rate} & Scheduled: Initial = 0.0003, & Scheduled: Initial = 0.0003, \\  & Factor = 0.9, Decay Steps = 10,000 & Factor = 0.9, Decay Steps = 10,000 \\  & (Exponential Decay) & (Exponential Decay) \\ \hline Optimizer & Adam & Adam \\ \hline Loss function & Binary\_crossentropy & Categorical\_crossentropy \\ \hline Metric & Accuracy & Accuracy \\ \hline \hline \end{tabular}
\end{table} 
Table 26: DNN model hyperparameters. Each head performs this attention calculation independently and then concatenates the results, as detailed in Equation (23) [95].