###### Abstract  
A speech emotion recognition (SER) model for noisy environments is proposed, by using four band-pass filtered speech waveforms as the model input instead of the simplified input features such as MFCC (Mel Frequency Cepstral Coefficients). The four waveforms retain the entire information of the original noisy speech while the simplified features keep only partial information of the noisy speech. The information reduction at the model input may cause the accuracy degradation under noisy environments. A normalized loss function is used for training to maintain the high-frequency details of the original noisy speech waveform. A multi-decoder Wave-U-Net model is used to perform the denoising operation and the Wave-U-Net output waveform is applied to an emotion classifier in this work. By this, the number of parameters is reduced to \(2.8\) M for inference from \(4.2\) M used for training. The Wave-U-Net model consists of an encoder, a 2-layer LSTM, six decoders, and skip-nets; out of the six decoders, four decoders are used for denoising four band-pass filtered waveforms, one decoder is used for denoising the pitch-related waveform, and one decoder is used to generate the emotion classifier input waveform. This work gives much less accuracy degradation than other SER works under noisy environments; compared to accuracy for the clean speech waveform, the accuracy degradation is \(3.8\%\) at \(0\) dB SNR in this work while it is larger than \(15\%\) in the other SER works. The accuracy degradation of this work at SNRs of \(0\) dB, \(-3\) dB, and \(-6\) dB is \(3.8\%\), \(5.2\%\), and \(7.2\%\), respectively. speech emotion recognition; emotion classification; SER; noisy environment; deep learning; speech waveform; denoising; band-pass filter; pitch; Wave-U-Net +
Footnote †: journal: Medical  
a generative noise model, where a randomly generated noise waveform is added to an utterance waveform at each epoch. The randomly generated noise waveform is obtained by multiplying a randomized frequency envelope by the frequency spectrum of white noise and then performing the inverse Fourier transform on the resultant frequency spectrum. An algorithm is used to extract a 6552 vector feature from an utterance. A DNN classifies the 6552 vector into four emotions. [PERSON] et al. (2021) [7] generates an MFCC vector from a noisy speech waveform at every 32 ms, accepts the MFCC matrix for an utterance as the input feature, classifies into four emotions after passing the input feature through several CNN (convolutional neural network) layers, an attention layer, and a DNN classifier. [PERSON] and [PERSON] (2021) [20] used two CNN blocks in series; one CNN block accepts the Short-time Fourier Transform (STFT) of a noisy speech as input and generates the cleaned-up STFT as output, the other CNN block accepts the cleaned-up STFT as input and performs the emotion classification. [PERSON] et al. (2022) [21] used a CNN block to rank the input features based on the noise resiliency, and then used only the high-ranked input features for the emotion recognition operation. After the above-mentioned work, [PERSON] et al. (2023) [22] cleaned up the low-ranked input features in terms of the noise resiliency by using GAN, and then used all the input features including the cleaned-up ones for the emotion recognition. The above-mentioned SERs for noisy environments tend to simplify the input speech waveform into rather compact features, such as MFCC, WFCC, and the 6552 vector for an utterance. Hence, a significant portion of information can be lost during the over-simplifying step, and the accuracy might be degraded especially under noisy environments. In this work, the entire time-domain waveform is used as the input to the deep learning model without generating the simplified input features from the waveform. Besides, the input waveform is converted into four band-pass waveforms in order to preserve the small-magnitude high-frequency components.