# Utilizing Machine Learning for Detecting Harmful Situations by Audio and Text
## 4 Methodology Description
### Methods Used for Classification via Audio
#### 4.2.2 Wav2 Vec 2.0 Pretrained Models  
The Wav2 Vec 2.0 pretrained models, developed by Facebook AI. [6], is a framework for self-supervised learning of vector representation from speech audio, pretrained in large quantities of audio data. This model attempts to recover a randomly masked portion of the encoded audio feature. The model consisted of three main modules. The first module was a feature encoder, composed of a 1-CNN encoder that downsamples the input raw waveform \(\mathcal{X}\) to a latent speech representation of 25 ms each \(\mathcal{Z}\) in T time steps. The second module was a contextualized encoder consisting of several transformer encoder blocks that transform latent representations \(\mathcal{Z}\) into contextualized representations \(\mathcal{C}\). In addition, a quantization module discretized the speech representation \(\mathcal{Z}\) into a finite set of quantized representations \(\mathcal{Q}\) by matching it with a code-book for selecting the most appropriate representation of the audio. Its objective was to identify these quantized representations of the masked features using the output of the contextualized network \(\mathcal{C}\) for each masked time step \(T\) using the contrastive loss function. After pre-training the unlabeled audio data, the model can be fine-tuned on the labeled data to be used for downstream tasks.  
In this study, we compared the accuracy of insulting and unsafe situation detection using text and audio content of spoken sentences in different situations. In particular, for detection through audio, we used both the Wav2 Vec 2.0, base model, called _Wav2 vec Base_ ([[https://huggingface.co/facebook/wav2](https://huggingface.co/facebook/wav2) vec2-base]([https://huggingface.co/facebook/wav2](https://huggingface.co/facebook/wav2) vec2-base), accessed on 15 March 2023, and a model that was fine-tuned on the SER task, called _Wav2 Vec for Emotions_. Their vector embedding sizes were 768 and 1024, respectively. In addition, we fine-tuned both models using our dataset.  
The Wav2 Vec 2.0 Emotion pretrained model ([[https://huggingface.co/ehalabres/wav2](https://huggingface.co/ehalabres/wav2) vec2-lg-xlsr-en-speech-emotion-recognition]([https://huggingface.co/ehalabres/wav2](https://huggingface.co/ehalabres/wav2) vec2-lg-xlsr-en-speech-emotion-recognition), accessed on 15 March 2023) was a fine-tuned model based on the Wav2 Vec 2.0 xlsr-53 model ([[https://huggingface.co/jonatasgrosman/wav2](https://huggingface.co/jonatasgrosman/wav2) vec2-large-xlsr-53-english]([https://huggingface.co/jonatasgrosman/wav2](https://huggingface.co/jonatasgrosman/wav2) vec2-large-xlsr-53-english), accessed on 15 March 2023), which was trained using the audio dataset [63] currently consisting of 7335 hours of transcribed speech in 60 languages. The Wav2 vec 2.0 emotion pretrained model used in this study is a model based on the Wav2 Vec 2.0 model and was fine-tuned for emotion recognition on the RAVDESS dataset [64], a multimodal database of emotional speech and song that contains 1440 hours of samples in eight different emotion classes, recording professional actors, in English.  
In addition, we fine-tuned the Wav2 Vec 2.0 base and Wav2 Vec 2.0 emotion pretrained models using our dataset. During the fine-tuning process, we extracted the context representation of our data from the pre-trained models, starting with an average pooling layer calculating an averaged vector according to the time dimension, followed by a fully connected layer with the tanh activation function. Finally, a fully connected layer was used for the classification task. Because the Wav2 Vec 2.0 model was used as a feature extractor, the weights of the feature encoder module of the pre-trained model were not changed during the fine-tuning process. This fine-tuning architecture was inspired by [65] because of its similarity to our task and its ability to achieve satisfactory results in their tasks. The hyper-parameters used for fine-tuning are listed in Table 4.