## 3 Materials and Methods
### Proposed Model  
This study's methodology centers on the construction of an ensemble model that combines transformer, CNN, and LSTM architectures. Transformer models use self-attention mechanisms to extract contextual features from input sequences, CNN filters to extract local temporal features from input sequences, and LSTM models to infer long-term dependencies from input sequences using recurrent connections. Therefore, in the context of this study, the transformer self-focuses on the interconnectedness of the input elements regardless of the sequence, the CNN layers identify the local audio feature patterns, and the LSTM layers capture and learn the long-term dependencies and temporal relationships within the audio sequences. LSTM layers use a series of gates (input, forget, and output gates) and a memory cell to selectively retain, update, or forget information from previous time steps, enabling them to effectively model and learn from long-term dependencies in the input sequences. For our study, the outputs of these three models were then merged and input into a dense final layer for classification, as shown in Figure 6. Thus, in our model, these architectures are combined to determine a feature of the input sequence that accounts for the contextual, local temporal, and long-term dependencies. The Softmax activation function in the dense layer classifies the combined feature representation to produce the final output. The transformer model features three transformer block layers containing multi-head self-attention layers and feed-forward neural networks (FFNNs). The multi-head self-attention layers embed 64 units and 8 heads, and the FFNNs have 64-unit hidden layers with rectified linear unit (ReLU) activation functions. The CNN model comprises 4 Conv1D layers with 64 filters and 3 kernel sizes, with the ReLU activating each Conv1D layer. The flattened layer then flattens the last Conv1D layer output, which comprises three 64-unit layers. The first two LSTM layers then return the sequences, and the last layer returns to the final hidden state. Then, the final output for emotion classification is generated by concatenating the outputs of the transformer, CNN, and LSTM architectures and running them through a dense final layer comprising six to eight units and a Softmax activation function. (1) Transformer Block: Transformer DL models use self-attention mechanisms to focus on different words in the input sequence when producing an output, with the transformer block using multi-head self-attention to focus on different positions and understand the data. FFNNs and dropout layers reduce overfitting with layer normalization, stabilizing the learning process [37]. Matrices W_Q, W_K, and W_V are subject to training and can be modified by the learning process. Furthermore, the attention scores (\(S\)) can be calculated by multiplying the query and key matrices by the square root of their dimensions using the following equation:  
\[S=QK^{*}T/sqrt(d\_k). \tag{4}\]  
where W denotes SoftMax (\(S\)), which yields the attention weights, and the attention layer output is a weighted sum of the value matrices: Z = WV. The FFNN in our model comprises a nonlinear activation function and two linear transformations, W_1, b_1, W_2, and b_2. These parameters denote the adjustable weights and biases of the attention layer. The FFNN outputs were connected by adding the input to the output using the following equation:  
\[\mathrm{Y}=\mathrm{X}+\mathrm{F}(\mathrm{X}). \tag{5}\]  
Layer normalization is the process by which the feature dimension input is normalized according to the attention layer and FFNN output, using the following equation:  
\[\mathrm{Y}=(\mathrm{X}-\mathrm{mean}(\mathrm{X}))/\mathrm{std}(\mathrm{X}), \ ^{*}\ \mathrm{Gamma}+\mathrm{beta}(\mathrm{trainable}\ \mathrm{scale}\ \mathrm{and}\ \mathrm{shift}\ \mathrm{parameters}).\]  
The transformer block in this model comprises an architectural component that accepts a feature vector of size (192, 1) as the input (see Table 2).