## 1 Introduction
For example, in reference [24], time-frequency information in the context of the Mel spectrogram was analyzed using a CNN. In addition to the above problems, The CNN deep features may contain redundancy, which may reduce recognition accuracy. To solve this problem, researchers have used various methods to optimize or select features. Traditional optimization or dimension-reduction methods include principal component analysis (PCA) [25] and linear discriminant analysis (LDA) [26], which are linearly dependent, and are widely used for SER. However, the non-linear feature selection algorithms perform better than linear feature selection algorithms [2]. For example, adversarial auto-encoder (AE), which is a linear feature selection algorithm, reduced the dimensions, optimized the features, and improved accuracy compared to PCA and LDA [27]. In this paper, a stacked sparse autoencoder (SSAE) is used to refine the deep features of a CNN: that is, multiple sparse autoencoders (SAEs) are superimposed to form a deep model structure with a strong ability to compute complex functions. The features extracted by multiple layers are more vivid and representative than those extracted by an SAE [28]. Different spectrograms can provide different types of speech emotion information. Instead of just using only a single type of information, we implemented a fusion strategy to combine multiple information sources. The Mel spectrogram and IMel spectrogram of the preprocessed speech are sent to two channel structures CNN separately, forming a dual-channel complementary structure. A CNN is used to extract one-dimensional features from the two-dimensional spectrograms of each of the two channels. The first channel extracts the deep features of the Mel spectrogram and highlights the low-frequency information. The second channel extracts the deep features of the IMel spectrogram and highlights the high-frequency information. The spliced features are then sent to an SSAE nonlinear structure to perform feature optimization. Finally, a softmax layer is used to identify the output. To summarize, the main improvements of this paper are as follows. 1. The IMel spectrogram, which is the complement of the Mel spectrogram, that focuses on the high-frequency part of the signal, is proposed. It can supplement the Mel spectrogram focusing on the low-frequency part of the signal. 2. The CNN-SSAE network, to extract deep optimized features of the spectrogram, is proposed. The deep features extracted by CNN often contain redundancy information that affects the recognition. In order to obtain the essential information of the features, this paper adopts SSAE to learn more abstract and representative features. 3. To improve the recognition rate, a dual-channel structure based on the Mel and IMel spectrograms is proposed. A single channel based on the Mel spectrogram is incomplete, and two channels are complementary. The structure of this paper is as follows. Section 2 introduces the related research. In Section 3, the architecture of SER is described. Section 4 describes the generation of the IMel spectrogram, and provides theoretical analysis. Section 5 presents the CNN-SSAE neural network for extracting deep optimized features. Section 6 introduces the datasets, features, experimental settings, and evaluation scenarios. Section 7 presents the conclusions and discusses the future research directions of SER.