## 2 Proposed Deep Learning Model  
The model proposed in this work accepts a noisy waveform (\(x\)) of an utterance as input and generates the probabilities of four emotions as output. The model consists of a block of four band-pass filters, a Wave-U-Net, and an emotion classifier (Figure 1). The passbands of the band-pass filters are 0\(\sim\)1 kHz, 1 kHz\(\sim\)2 kHz, 2 kHz\(\sim\)4 kHz, and 4 kHz\(\sim\)8 kHz. Three 401-tap FIR (finite impulse response) low-pass filters with bandwidths of 1 kHz, 2 kHz, and 4 kHz are used to generate the four band-pass waveforms. The Wave-U-Net consists of an encoder, a 2-layer LSTM block, six decoders, and skip-nets which deliver the intermediate outputs of the encoder to the decoders. The six decoders generate four cleaned-up waveforms (\(\hat{y}_{1}\), \(\hat{y}_{2}\), \(\hat{y}_{4}\), and \(\hat{y}_{8}\)), a pitch-related waveform(\(\hat{y}_{p}\)), and a waveform (\(\hat{y}_{emo}\)) which will be the input to the emotion classifier. The emotion classifier generates the probabilities of four emotions (angry, happy, sad, and neutral). The sample rate of the noisy input waveform (\(x\)) is 16 kS/s. The waveform is applied to the band-pass filter, and the four band-pass filter outputs are up-sampled to 64 kS/s before being applied to the encoder. All the six decoder outputs are down-sampled back to 16 kS/s. The encoder is composed of five CNN layers in series connection (Figure 2a); each CNN layer consists of a series connection of a down-sampling Conv1D layer with K = 12 and S = 4, GELU, a Conv1\(\times\)1 layer, and GLU; K is the kernel size of the 1D CNN and S is the stride. The shapes of the noisy waveform (\(x\)) and the band-pass filter output (\(x_{bpf}\)) are [T] and [T,4], respectively; T is the number of sample points of an utterance at the sample rate of 16 kS/s. The encoder input (\(x^{\prime}_{bpf}\)) has the shape of [4T, 4]. The output channel size of each CNN layer increases from 20, 40, 80, 160, and 320, as the encoding step proceeds. The output of each CNN layer is sent to both the upper-level CNN layer and the skip-net. The encoder output (\(z\), [T/256, 320]) is sent to the 2-layer bi-directional LSTM to generate the same-shape output (\(z^{\prime}\)). Figure 1: The proposed model for speech emotion recognition, a multi-decoder Wave-U-Net. The LSTM output (\(z^{\prime}\)) is sent to six decoder blocks. Each decoder block consists of a Conv1\(\times\)1, a decoder, and a 4\(\times\) down-sampler, and a skip-net. To maintain the same number of trainable parameters for the encoder and the six decoders, the input channel size of a decoder is around 1/\(\sqrt{6}\) times the output channel size of the corresponding encoder. The Conv1\(\times\)1 layer of each decoder block reduces the channel size from 320 to 131. Each decoder of the decoder block is composed of 5 CNN layers in series connection; each CNN layer consists of a series connection of a Conv1\(\times\)1 layer followed by GLU and an up-sampling TransposedConv1D layer with K = 12 and S = 4, and GELU. GELU is not added at the last CNN layer of the decoder. The input channel size of the CNN layers of a decoder is 131, 65, 33, 16, and 8, respectively. The shape of the final CNN layer of the decoder is [4\({}^{\prime}\)T]; this is down-sampled by 4\(\times\) at the final stage of the decoder block to generate \(\hat{y}\) with the shape of [T]. The skip-net of each decoder block consists of five Conv1\(\times\)1 layers; they reduce the output channel size to compensate for the channel size difference between the encoder and the decoder. Figure 2: Detailed structure of the proposed speech emotion recognition model shown in Figure 1 (a) encoder and decoder block with skip-networks; (b) emotion classifier. The emotion classifier accepts the input \(\hat{y}_{emo}\) with the shape of [T] and generates four probabilities (\(\hat{p}_{emo}\), [4]) for the emotions.