# Speech Emotion Recognition Using Deep Learning Transfer Models and Explainable Techniques
###### Abstract  
This study aims to establish a greater reliability compared to conventional speech emotion recognition (SER) studies. This is achieved through preprocessing techniques that reduce uncertainty elements, models that combine the structural features of each model, and the application of various explanatory techniques. The ability to interpret can be made more accurate by reducing uncertain learning data, applying data in different environments, and applying techniques that explain the reasoning behind the results. We designed a generalized model using three different datasets, and each speech was converted into a spectrogram image through STFT preprocessing. The spectrogram was divided into the time domain with overlapping to match the input size of the model. Each divided section is expressed as a Gaussian distribution, and the quality of the data is investigated by the correlation coefficient between distributions. As a result, the scale of the data is reduced, and uncertainty is minimized. VGGish and YAMNet are the most representative pretrained deep learning networks frequently used in conjunction with speech processing. In dealing with speech signal processing, it is frequently advantageous to use these pretrained models synergistically rather than exclusively, resulting in the construction of ensemble deep networks. And finally, various explainable models (Grad CAM, LIME, occlusion sensitivity) are used in analyzing classified results. The model exhibits adaptability to voices in various environments, yielding a classification accuracy of 87%, surpassing that of individual models. Additionally, output results are confirmed by an explainable model to extract essential emotional areas, converted into audio files for auditory analysis using Grad CAM in the time domain. Through this study, we enhance the uncertainty of activation areas that are generated by Grad CAM. We achieve this by applying the interpretable ability from previous studies, along with effective preprocessing and fusion models. We can analyze it from a more diverse perspective through other explainable techniques. speech emotion recognition; explainable model; deep learning; YAMNet; VGGish; audible feature +
Footnote †: journal:  
and the performed classification. Applying appropriate preprocessing can remove these unnecessary elements while increasing the quality of voice data [4]. Additionally, appropriate feature extraction methods are required for analyzing speech complexity and diversity [5]. A spectrogram shows the shape of the extracted features, which allows visual inspection of the frequency component changes over time. Moreover, it can be analyzed by viewing it as an image and then applying a computer vision technique [6]. In a spectrogram study, an explainable model can be applied to determine the relationship between the input and output data to identify which time-frequency domain influences the results of the classification model. When the results of a classification model are analyzed using an explainable model, the reliability of the former is increased, and its performance can be improved by additional research and analysis [7]. This study converted acquired speech signals into spectrogram images for analysis in the time-frequency domain. Unnecessary elements (such as noise, silence, and disconnection, etc.) that could lead to an incorrect decision-making process through the classification model were processed. Such elements are inevitably created in the process of acquiring speech. Instead of manual removal, an algorithm using Gaussian distribution and correlation coefficients was applied to exclude unnecessary divisions, which reduced the learning time and computational resource consumption. Three datasets were used for emotion classification to ensure that the classification model could be generalized.