# Efficient and Robust Arabic Automotive Speech Command Recognition System
## 3 Methodology
### Classifier Selection
#### 3.3.3 Transformers  
Transformers, a specific kind of deep neural network with an attention mechanism, were originally developed for natural language processing [37] to build large language models. They have rapidly gained prominence in various fields, including audio classification [38]. The core innovation behind transformers is their multi-head self-attention mechanism [39], which enables the model to weigh the importance of different parts of the input sequence, capturing useful global-context information and complex dependencies [40]. Moreover, it can work with variable input sequence lengths without traditional recurrent architectures' constraints. Furthermore, in audio classification, transformers excel due to their ability to model long-range dependencies in audio signals. Unlike traditional convolutional or recurrent neural networks, transformers can process entire audio sequences simultaneously, allowing them to capture local and global patterns in the data. This capability is particularly beneficial for tasks where the context and temporal structure of the audio are crucial, such as sound-event detection, speaker identification, and audio classifying across diverse conditions and applications [41]. Figure 4: Proposed BiLSTM-CNN Architecture. Pre-trained transformer models like Wave to Vector (wav2 vec2) [42], Hidden-Unit BERT (HuBERT) [43], Deep Speech [44], and Whisper [45] have been fine-tuned for ASR tasks, demonstrating state-of-the-art performance on various benchmarks. These models leverage large-scale, self-supervised learning on massive datasets, enabling them to generalize effectively to both new and noisy audio. Therefore, in our study, we will fine-tune the Wav2 vec2 models on our dataset and conduct a comparative analysis between its results and those of the machine- and deep-learning models we built. Fine-Tuning the Wav2 vec2 Model  
Wav2 Vec2 is a speech recognition system that relies on supervised learning but can alternatively adapt to unsupervised learning methods developed by Facebook [42]. The choice of Wav2 vec2 over other pre-trained transformer models is motivated by the fact that Wav2 Vec2 is an advanced speech recognition model that leverages a transformer-based architecture to learn the speech structure from raw audio. Unlike traditional models that require large amounts of labeled data, Wav2 Vec2 uses a self-supervised learning approach, allowing it to learn meaningful speech representations from unlabeled audio by predicting masked segments of the audio waveform, similar to how models like Bidirectional Encoder Representations from Transformers (BERT) [46] predict masked words in the text. Furthermore, the model showed a promising result in the experiments. Through training the model with just 10 min of transcribed speech and 53,000 h of unlabeled speech, wav2 vec 2.0 enables speech recognition models at a word-error rate (WER) of 8.6% on noisy speech and 5.2% on clean speech on the standard LibriSpeech benchmark [42]. The Wav2 Vec2 model starts by feeding the original audio waveform through a convolutional neural network (CNN) to extract initial feature representations. These representations are latent audio features, each spanning 25 ms, which are subsequently processed by a transformer network. This network leverages self-attention, enabling the model to capture important dependencies and contextual nuances throughout the entire audio sequence. This design empowers Wav2 Vec2 to effectively model extensive relationships within audio data, enhancing its accuracy and resilience in diverse speech recognition tasks, especially in challenging environments with noise or limited resources [42]. Fine-tuning is a common technique for raising a pre-trained model's performance on a specific task by taking the pre-trained model and training it further on a smaller dataset that is specific to the proposed use case.