## 2 Proposed Deep Learning Model
The emotion classifier is composed of four Conv1D layers with K = 12 and S = 4 followed by the Global average layer, two linear layers, and a softmax layer (Figure 2b). The Global average layer performs the time-averaging operation and converts the input with the shape of [T/256, 256] to the output with the shape of [256]. Table 1 lists the number of parameters of each block of the proposed model. The encoder, the LSTM, and the decoder blocks use 1.1 M, 1.2 M, and 1.7 M parameters, respectively. For training, all the blocks of Figure 1 are activated and 4.2 M parameters are used. For inference, the decoder blocks including the five decoders on the upper left-hand side of Figure 1 are turned off and only 2.8 M parameters are used. The model parameters are trained to fit the five waveforms (\(\hat{y}_{1}\), \(\hat{y}_{2}\), \(\hat{y}_{4}\), \(\hat{y}_{8}\), and \(\hat{y}_{p}\)) and an emotion probability vector (\(\hat{p}_{emo}\)) to the ground-truth counterparts. Among the ground-truth counterparts, the emotion probability vector (\(p_{emo}\)) is a one-hot vector ([4]) corresponding to the ground-truth clean speech waveform. The ground-truth band-pass waveforms (\(y_{1}\), \(y_{2}\), \(y_{4}\), and \(y_{8}\)) are generated by applying the three FIR low-pass filters to the ground-truth clean speech waveform (Figure 3). The ground-truth pitch-related waveform (\(y_{p}\)) is generated from the pitch mark timing data, which are provided by running REAPER [24] on the ground-truth clean speech waveform. Because the discrete-time pitch mark timing is difficult to train with the deep learning model, the pitch mark timing is replaced by a continuous-time pitch-related waveform (\(y_{p}\)); the pitch-related waveform is a sinusoidal wave during the voiced period and is 0 during the unvoiced period. Because the pitch mark timing is not uniform in time, the ground-truth pitch-related waveform (\(y_{p}\)) is a sine wave with non-uniform periods; the rising zero-crossing time of the sine wave matches the pitch mark timing and the amplitude of the sine wave follows the envelope of the ground-truth clean speech waveform. The envelope follows the absolute value of the ground-truth clean speech waveform with the exponential rise and fall decays; the time constant is 5 ms. The loss function (Equation (1)) is the sum of the wave loss function (Equation (2)) and the emotion loss function (Equation (3)). The wave loss function is a normalized L1 loss averaged over N utterances and summed over five waveforms (1, 2, 4, 8, and \(p\)); the waveform is normalized by the standard deviation and \(\epsilon\) is \(10^{-4}\). The emotion loss function is a cross-entropy function. \[L_{\text{total}}(\hat{y},y) =L_{\text{wave}}(\hat{y},y)+L_{\text{emo}}(\hat{y},y) \tag{1}\] \[L_{\text{wave}}(\hat{y},y) =\frac{1}{N\times T}\sum_{m\in M}\sum_{n=1}^{N}\sum_{t=1}^{T} \frac{|g_{m,n}(t)-y_{m,n}(t)|}{\sigma_{y_{m,n}}+\epsilon},\quad\text{where }M=\{1,2,4,8,p\}\] (2) \[L_{\text{emo}}(\hat{p},p) =-\frac{1}{N}\sum_{n=1}^{N}\sum_{c=1}^{C}p_{n,c}\log(\hat{p}_{n,c}) \tag{3}\] \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Step**} & \multicolumn{4}{c}{**Component**} & \multirow{2}{*}{**Total**} \\ \cline{2-2} \cline{4-5}  & **Encoder** & & & **LSTM** & **Decoder Blocks** & **Emotion Classifier** \\ \hline Training & 1.1 & 1.2 & 1.7 & 0.2 & 4.2 \\ Inference & 1.1 & 1.2 & 0.3 & 0.2 & 2.8 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Number of parameters of each component of the proposed model in million units.