## 3 Materials and Methods
### Proposed Model
It consists of a multi-head self-attention mechanism and an FFNN, each of which employs residual connections and layer normalization. In addition, the multi-head self-attention's embed_dim and num_heads parameters represent the size of the input embeddings and number of attention heads. Each attention head individually processes the input, thereby allowing the model to concurrently learn different types of information from a singular input sequence. In this mechanism, query_dense, key_dense, and value_dense are the dense layers that transform the inputs into their corresponding query, key, and value vectors. These vectors were further separated into different heads using the separate_heads method, ensuring parallel and independent computations for each head. The call method subsequently orchestrates the computation flow of the self-attention mechanism by calling the previous components sequentially. After these computations, the dense combined head layer merges the outputs from all attention heads back into the original embedding dimension. The transformer block also features an FFNN, characterized by ff_dim, which denotes the size of its hidden layer. To prevent overfitting, two dropout layers with a rate defined by the hyperparameter \"rate\" were employed. The feed-forward network (FFN) in the transformer block consists of two dense layers: the first one applies a \"relu\" activation function, and the second one, which has the same size as embed_dim, applies no activation function. The FFN is applied to the output of the multi-head attention mechanism, which computes the attention weights between all pairs of positions in the input sequence using multiple attention heads. The purpose of the FFN is to process the concatenated output from different attention heads, allowing the model to capture more complex dependencies and transformations. After the FFN, the output traverses another dropout layer and undergoes a residual connection, followed by normalization using a second normalization layer. In summary, the transformer block transforms the input embedding, outputting a transformed embedding of the same size with tunable hyperparameters such as the number of attention heads and the size of the hidden layer in the FFN. (2) CNN: In parallel with the transformer block, a series of Conv1D layers are employed in our model to extract local temporal features directly from the input acoustic features. The Conv1D layers consist of 64 filters with a kernel size of 3. The \"same\" padding technique is used to ensure that the output has the same width as the input. The ReLU activation function is applied element-wise to introduce nonlinearity to the model. Each filter (f) in the Conv1D layers is a vector of weights of size (k), where k denotes the kernel size. The output of the convolutional layer at position i is determined using the following equation:  
\[(\mathrm{x}*\mathrm{f})(\mathrm{i})=\mathrm{sum}(\mathrm{x}[\mathrm{i}+ \mathrm{j}]*\mathrm{f}[\mathrm{j}])\ \mathrm{for}\ \mathrm{j}=0,\dots,\mathrm{k}-1. \tag{6}\]  
If padding is used, the input sequence is expanded with zeroes before the filters are applied. Additionally, when a nonlinear activation function, such as ReLU, is employed, it is implemented on every individual element of the output of the convolutional layer. To reiterate, the padding technique known as \"same\" is used to pad the sides of the input to achieve a width that matches that of the output; the ReLU activation function then introduces nonlinearity to the model by producing an input value as an output if it is positive and as zero if it is negative; and the CNN layers then use filters to obtain local features from the input vectors through a convolution.