## 5 Proposed Models
### Transformer-Enhanced CNN-RNN (TE-CNN-RNN)
#### 5.1.2 Multi Transformer Blocks  
At the heart of our TE-CNN-RNN framework lies the Multi Transformer Blocks, which utilize a Multi-Head Attention (MHA) mechanism for parallel processing of input data. This architecture is crucial in capturing diverse dependencies, significantly enriching the model's ability to interpret complex data streams.  
To overcome the limitations of traditional CNN-RNN architectures, we integrated Multi Transformer Blocks into our TE-CNN-RNN model. These architectures often struggle with temporal dynamics and long-range dependencies in sequential data. The Multi-Head Attention mechanism within these blocks allows our model to focus dynamically on various segments of the input sequence. This feature is particularly crucial in applications like traffic forecasting, where diverse and rapidly changing factors can significantly influence patterns. The MHA mechanism enhances the model's feature representation and decision-making capabilities by simultaneously processing different data aspects, providing a more accurate and robust framework for understanding and predicting complex dynamic systems. The Multi Transformer Blocks encompass the following components:  
Multi-Head Attention (MHA)  
Multiple attention heads are utilized to process distinct segments of the input sequence in parallel, enhancing the model's ability to recognize network traffic data patterns. Given an input sequence INT \(X=(X_{0},X_{1},\dots,X_{n})\), the Multi-Head Attention mechanism is applied:  
\[\text{MHA}(X)=\text{Concat}(\text{head}_{1},\dots,\text{head}_{h})W^{O} \tag{1}\]  
Figure 2: Transformer-enhanced CNN-RNN (TE-CNN-RNN).  
where  
\[\text{head}_{i}=\text{Attention}(XW_{i}^{Q},XW_{i}^{K},XW_{i}^{V}) \tag{2}\]  
\(W^{Q}\), \(W^{K}\), \(W^{V}\), and \(W^{O}\) are the parameter matrices for query, key, value, and output, respectively. The INT referred to earlier is the original data, i.e., the regular data before feature extraction, as explained in Section 5.3.1.  
Dropout and residual connections are integrated within our architecture to prevent overfitting and ensure that important information is retained throughout the network, thus promoting smoother gradient flow and enhancing generalization. These connections combine the MHA's output with the input sequence, preserving vital data and enabling a smoother gradient flow.  
\[X^{\prime}=\text{Dropout}(\text{MHA}(X))+X \tag{3}\]  
Layer normalization standardizes the data across features within each sample, stabilizing training dynamics and accelerating convergence. Applied before and after the feed-forward network, it normalizes the data across features, ensuring a consistent mean and variance. This stabilizes gradient propagation during training and prevents any single feature from dominating the learning process, leading to more stable updates.  
\[X_{\text{norm}}=\text{LayerNorm}(X^{\prime}) \tag{4}\]  
The feed-forward network (FFN) deepens the architecture, enabling the model to interpret the refined data from the MHA and other preprocessing steps further. This is essential for making accurate final predictions or classifications. It consists of two dense layers with ReLU activation in between, aimed at processing the sequence further to uncover complex data relationships.  
\[\text{FFN}(X_{\text{norm}})=\text{ReLU}(X_{\text{norm}}W_{1}+b_{1})W_{2}+b_{2} \tag{5}\]  
with \(W_{1}\), \(W_{2}\), \(b_{1}\), and \(b_{2}\) being the weights and biases of the FFN layers.  
The Output  
Finally, the architecture applies another layer normalization step to the FFN's output, yielding the finalized feature set for subsequent fusion.  
\[X_{\text{final}}=\text{LayerNorm}(\text{FFN}(X_{\text{norm}})+X_{\text{norm}}) \tag{6}\]