# Speech Emotion Recognition Using Deep Learning Transfer Models and Explainable Techniques
## 2 Related Work
### Studies Applying Deep Learning and Explainable Methods to Spectrograms  
This section presents various studies in which deep learning was applied to 2D spectrogram images, using explainable models. The model's decision-making process was verified by applying explainable techniques to increase its reliability, and activation areas were also used as data.  
[PERSON] [14] converted speech data in the IEMOCAP dataset into spectrogram images and extracted feature maps using AlexNet. The feature maps processed attention using the tanh and softmax activation functions to highlight the essential feature channelsin the attention layer. They identified that the model, applied by CNN and LSTM, achieved a WA of 68.8% and UA of 59.4%; however, the accuracy was effectively improved to a WA of 70.4% and a UA of 63.9% when attention was applied. The focus area of the model was visually confirmed using Grad CAM. [PERSON] [15] proposed recognizing a speaker's place of local origin using the VCTK dataset. Speech data were converted into spectrogram images and trained using CNN models. The essential areas created using Grad CAM were combined with a spectrogram to create new feature vectors. These feature vectors were applied to various machine learning algorithms (MLAs) to classify a speaker's place of local origin. The model performances were evaluated using the unweighted average recall (UAR) and macro average accuracy (MAA) and compared with those of other methods, such as the speech wave input and other MLA methods. SVM was measured as 0.348 in MAA and 0.340 in UAR, GaussianNB was measured as 0.262 in MAA and 0.262 in UAR, and passive aggressive was measured as 0.351 in MAA and 0.356 in UAR. [PERSON] [16] conducted a study on acoustic scene classification, in which sound signals were analyzed to identify the environment and scene to manage and analyze the surroundings. This study used the dataset employed in the DCASE 2016 challenge. Spectrogram images converted into speech signals were trained using ResNet to classify three different environments and identify the areas that influenced the classification decision. The classification achieved an accuracy of 91.82% and the time-frequency area related to the classified class is highlighted using Grad CAM. This highlighted area is then converted into a sound signal for confirmation. [PERSON] [17] used PCG signal data from Kaggle to train phonocardiogram signals with various hyperparameter values and 2D-CNN models and to classify normal and abnormal sounds. That model was evaluated with an accuracy of 86%, and the decision of the model to classify normal and abnormal sounds in the time domain of the heartbeat was confirmed using Grad CAM for the results. [PERSON] [18] conducted a study to predict the recovery of patients who underwent thyroid cancer surgery by converting their speech sounds before and after surgery into spectrograms. The training involved the use of EfficientNet and LSTM, with patients and vocal data provided by DIRAMS. The results showed that the trained model achieved an evaluation score of 0.822 by AUROC. Additionally, Grad CAM was used to confirm the area affected in the time-frequency domain corresponding to the classification.  
In a study that applied deep learning and explanatory techniques to the spectrogram, the decision-making process of the model was mainly confirmed using only Grad CAM. However, since the information that can be obtained with a heat map is limited, it is advantageous to use more diverse models and aspects.