# A Feature Fusion Model with Data Augmentation for Speech Emotion Recognition
###### Abstract  
The Speech Emotion Recognition (SER) algorithm, which aims to analyze the expressed emotion from a speech, has always been an important topic in speech acoustic tasks. In recent years, the application of deep-learning methods has made great progress in SER. However, the small scale of the emotional speech dataset and the lack of effective emotional feature representation still limit the development of research. In this paper, a novel SER method, combining data augmentation, feature selection and feature fusion, is proposed. First, aiming at the problem that there are inadequate samples in the speech emotion dataset and the number of samples in each category is unbalanced, a speech data augmentation method, Mix-wav, is proposed which is applied to the audio of the same emotion category. Then, on the one hand, a Multi-Head Attention mechanism-based Convolutional Recurrent Neural Network (MHA-CRNN) model is proposed to further extract the spectrum vector from the Log-Mel spectrum. On the other hand, Light Gradient Boosting Machine (LightGBM) is used for feature set selection and feature dimensionality reduction in four emotion global feature sets, and more effective emotion statistical features are extracted for feature fusion with the previously extracted spectrum vector. Experiments are carried out on the public dataset Interactive Emotional Dyadic Motion Capture (IEMOCAP) and Chinese Hierarchical Speech Emotion Dataset of Broadcasting (CHSE-DB). The experiments show that the proposed method achieves 66.44% and 93.47% of the unweighted average test accuracy, respectively. Our research shows that the global feature set after feature selection can supplement the features extracted by a single deep-learning model through feature fusion to achieve better classification accuracy. speech emotion recognition; data augmentation; feature selection; multi-head attention; features fusion +
Footnote †: journal: Medical Image and Vision  
emotion tasks, the collection of speech emotion data is limited. At present, the amount of dataset data in the field is relatively small [4; 5; 6], up to tens of thousands. The number of dataset samples is not as large as Imagenet [7] and Audioset [8], and is still not completely unified between performances, affective models and collection methods. In view of the lack of data, data augmentation is a convenient and feasible method. The commonly used speech data augmentation methods are speed disturbance [9], pitch adjustment [10], the addition of noise [11], increasing the amount of data by changing the audio rate, fundamental frequency, or adding noise to the original voice. In addition, [PERSON] et al. applied Generative Adversarial Networks (GANs) to generate spectrograms, enhanced the training data of samples, and conducted experiments on IEMOCAP and FEEL-25k emotion database [12]. [PERSON] et al. used the Vocal Tract Length Perturbation (VTLP) method to enhance the speech emotion classification model [13]. The authors in [14] proposed pre-training neural networks named Pretrained Audio Neural Networks (PANNs) for audio scene classification, using two data enhancement methods, Mixup [15] and SpecAugment [16]. Researchers have constantly explored audio data enhancement methods, from traditional methods to modern methods such as neural networks. We have also explored and migrated the methods suitable for speech synthesis, which can improve this problem. Deep learning accelerates the development of speech emotion. However, as the first step in speech signal processing-feature extraction, various manually designed low-level features are still widely used in SER [17; 18; 19]. The discovery of a speech acoustic feature that can effectively carry speech emotion information has become the key to the improvement of SER. At present, Mel Frequency Cepstral Coefficient (MFCC) and Log-Mel spectrograms are widely used in SER research [20; 21; 22] because that they can better describe the time-frequency correlation of emotional details [23].