# Speech Emotion Recognition through Hybrid Features and Convolutional Neural Network
## 1 Introduction  
Speech is the natural and widespread method of human communication and carries both paralinguistic and linguistic information. The linguistic information includes the context and language of the speech, while paralinguistic information includes the gender, emotions, age, and other unique attributes of the human. Several studies have shown that audio signals can be a simple mean to establish a connection between machines and humans [1]. Nonetheless, this involves the machine becoming familiar with the human voice so that the machine can predict the emotion, similar to humans. This has led to a growing interest in the area of SER, which involves identifying the emotions of speakers from their voices. SER is a crucial area of research that has various applications in the field of call centers [2, 3], human-computer interaction (HCI) [4], automatic translation systems, driving a vehicle [5, 6], and in healthcare, where patient emotional states are identified through voice, and necessary facilities are provided [7, 8]. However, as individuals have differentspeaking styles, and cultural backgrounds, the selection of relevant acoustic features becomes complex and challenging. Currently, voice features used for SER are categorized into spectral, continuous (formants, energy, pitch), Teager Energy Operator, and qualitative (voice quality) features [9]. Nevertheless, these expert-driven acoustic features typically rely on the experience of area experts, representing low-level features that are not helpful for recognizing emotions in complex situations. In summary, the major drawbacks of expert-extracted features are:  
* The ability to recognize emotional declines in complex situations, such as inter-speaker differences, variations in expressions, and environmental factors [10];
* The expert-driven features require significant time, money, and human expertise in order to train ML classifiers to enhance the efficiency [11];
* Currently, there is no established algorithm available to extract the ideal features to identify emotions [12]. In order to address the challenges mentioned above, it is necessary to implement effective techniques that can extract emotionally relevant and significant features for SER. To this end, various studies have proposed techniques that involve automatic feature extraction from voice signals. For example, a study [13] utilized a single-layer CNN to derive automatic feature, while another study [14] implemented a CNN with two convolutional layers (CLs) followed by a Long Short-term Memory (LSTM) layer for a SER system. However, shallow architectures such as single-layer and double-layer CNNs may not be able to learn salient features. In Ref. [15], a deep CNN was utilized to derive the discriminative frequency features by employing a rectangular filter along with a customized pooling technique for the SER. The suggested deep CNN was trained on the derived features from audio data. In this study, a novel approach for SER is proposed, which combines the MFCCs and time-domain features derived from each audio signal in dataset. The proposed approach consists of four main components: (1) data collection (2) feature extraction, (3) model training, and (4) prediction, as shown in Figure 1. In the feature extraction stage, the traditional features including MFCCs, and time-domain are extracted. In the feature fusion stage, the extracted features are concatenated. In the final stage, a CNN model that comprises three 1D CLs, following an activation, dropout, and max-pooling layers, as well as a fully connected (FC) layer, is used for SER. To estimate the performance of methodology, three publicly datasets: Emo-DB, Surrey Audio-Visual Expressed Emotion (SAVEE), and The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) were employed. It is worth noting that the expression of emotion varies among different speakers due to factors such as gender, cultural background, and accent. The proposed method achieved better recognition performance for SER.