# Utilizing Machine Learning for Detecting Harmful Situations by Audio and Text
## 5 Experimental Result
### Results on Combined Text and Audio Inputs  
Our next set of experiments was designed to evaluate the accuracy of insulting and unsafe sentence detection when text and audio data were combined. It was assumed that adding audio with text would enhance the results of each of them separately, but not all the additions produced the expected results. \begin{table}
\begin{tabular}{l l l} \hline  & **Audio** & **FT Audio** \\ \hline RNN & 0.60 & - \\ Eight Emotions & 0.50 & - \\ FSFM & 0.60 & - \\ Base & 0.59 & 0.64 \\ Emotion & 0.63 & 0.65 \\ Random & 0.49 & - \\ \hline \end{tabular}
\end{table} 
Table 10: Accuracy of unsafe speech detection based on audio only. \begin{table}
\begin{tabular}{l l l} \hline  & **Audio** & **FT Audio** \\ \hline RNN & 0.41 & - \\ Eight Emotions & 0.35 & - \\ FSFM & 0.44 & - \\ Base & 0.45 & 0.43 \\ Emotion & 0.47 & 0.48 \\ Random & 0.32 & - \\ \hline \end{tabular}
\end{table} 
Table 11: Accuracy of classifiers for all three categories based on audio only. \begin{table}
\begin{tabular}{l l l} \hline  & **Audio** & **FT Audio** \\ \hline RNN & 0.60 & - \\ Eight Emotions & 0.48 & - \\ FSFM & 0.60 & - \\ Base & 0.61 & 0.64 \\ Emotion & 0.63 & 0.66 \\ Random & 0.51 & - \\ \hline \end{tabular}
\end{table} 
Table 9: Accuracy of insulting speech detection based on audio only. Tables 12-14 describe the experiment's results for various combinations of text and wave embedded vectors. The methods are compared to a baseline method that uses a randomly generated vector with a length of 768 sampled from the same distribution as the Wav2 Vec vectors (referred to as the random vector), which was used instead of the Wav2 Vec embedding vectors. with and without fine-tuning the embedding vectors. This challenge can be viewed as a classification problem and categorized into the following three classes: neutral sentences, sentences consisting of insulting content, and sentences indicating unsafe situations. We concentrated on situations relevant to children, particularly those with special needs. To solve this challenge and examine different classification schemes, we created a new dataset consisting of over 1981 Hebrew sentences, where for each sentence, both text and audio recordings were collected. We compared the classification accuracy of different state-of-the-art supervised and self-supervised methods. In particular, we used Heb-BERT embedding vectors for the text with and without fine-tuning and compared the accuracy of different ML methods running on the TF-IDF vectors with respect to their accuracy. For the audio content, we compared different variations of Wav2 Vec embedding vectors, with and without fine-tuning, using different classical ML methods and DL methods. In our experiments, we found that adding embedded wave information to the text information only slightly improved the overall accuracy. This may be because in unsafe situations, the main information conveyed was related to the words being spoken and not the way they are spoken or other characteristics of the audio. This may be especially true for a dialogue between people who are not close to family members. This study makes significant contributions to two research areas. First, it can help develop automated agents to assist children with special needs. Second, it may advance the research on content classification and threat detection using text and voice signals. We would like to expand on these aspects. There are some open issues that can be addressed in future studies. The first is the ability to recognize unsafe and insulting situations using only automated tools. Note that, in our study, Hebrew texts were produced manually based on audio. Observing whether audio could more significantly impact if an ASR was used to produce the text is interesting, given the fact that current Hebrew ASRs are not accurate. Another interesting expansion is to check for other possible unsafe situations.