# Improving Singing Voice Separation Using Curriculum Learning on Recurrent Neural Networks
## 4 Evaluation
### Experimental Results of MUSDB18 Dataset  
MUSDB18 [32] is a much larger dataset than MIR-1K and ccMixter. The training set consists of totally 100 songs, approximately 10 hours long, and the test set consists of 50 songs. This dataset is a multitrack format consisting of five streams that are divided into mixtures, drums, bass, rest of the accompaniment and vocals. The multitrack mixture is used as inputs, and only vocal sounds are extracted from the mixture. The sum of all the other instrument sounds is considered as instrument sounds in the experiments.  
\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**Instrument**} & \multicolumn{4}{c}{**Vocal**} \\ \hline \multicolumn{2}{c}{**Model**} & \(GSIR_{m}\) & \(GSAR_{m}\) & \(GNSDR_{m}\) & \(GSIR_{v}\) & \(GSAR_{v}\) & \(GNSDR_{v}\) \\ \hline \multirow{2}{*}{RNN} & baseline & 10.75 & 10.38 & 3.26 & 11.09 & 3.18 & 5.60 \\  & proposed & (\(\downarrow\))10.41 & \(\uparrow\)**11.30** & 3.53 & \(\downarrow\)**12.35** & \(\uparrow\)**3.73** & \(\uparrow\)**6.34** \\ \hline \multirow{2}{*}{U-Net} & baseline & 9.00 & 12.98 & 3.36 & 14.55 & 3.38 & 6.27 \\  & proposed & 9.25 & (\(\downarrow\))12.71 & 3.50 & (\(\downarrow\))14.50 & 3.85 & 6.68 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Separation performance comparison for ccMixter dataset. Notations are same as in Table 2.  
\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**Instrument**} & \multicolumn{4}{c}{**Vocal**} \\ \hline \multicolumn{2}{c}{**Model**} & \(GSIR_{m}\) & \(GSAR_{m}\) & \(GNSDR_{m}\) & \(GSIR_{v}\) & \(GSAR_{v}\) & \(GNSDR_{v}\) \\ \hline \multirow{2}{*}{RNN} & baseline & 10.75 & 10.38 & 3.26 & 11.09 & 3.18 & 5.60 \\  & proposed & (\(\downarrow\))10.41 & \(\uparrow\)**11.30** & 3.53 & \(\downarrow\)**12.35** & \(\uparrow\)**3.73** & \(\uparrow\)**6.34** \\ \hline \multirow{2}{*}{U-Net} & baseline & 9.00 & 12.98 & 3.36 & 14.55 & 3.38 & 6.27 \\  & proposed & 9.25 & (\(\downarrow\))12.71 & 3.50 & (\(\downarrow\))14.50 & 3.85 & 6.68 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison of separation performance for MIR-1K dataset by global normalized source to distortion ratio (GNSDR), global source to artifact ratio (GSAR) and global source to interference ratio (GSIR). “Instrument” columns with subscript “_m_” are the evaluation results of instrument sound extraction, and “Vocal” columns with subscript “_v_” are those of vocal sound extraction. The baseline models are stacked RNN (“RNN” row) and U-Net (“U-Net” row). The rows with “proposed” header are the models trained by the proposed curriculum learning method.  
Table 4 shows the results of the experiments performed on the MUSDB18 dataset [32]. The most significant increment was observed in \(GSIR_{m}\) in RNN, 1.69 dB, but there were decrement in \(GSAR_{m}\) and \(GSIR_{v}\) of U-Net. The overall performance measured by \(GNSDR\) values were all increased. The smallest and the highest results were 0.38 dB and 1.06 dB. These results show that the proposed method is also effective for large datasets as well.