# Improving Singing Voice Separation Using Curriculum Learning on Recurrent Neural Networks
## 4 Evaluation
### Experimental Results of MIR-1K Dataset  
Singing voice separation experiments with simulated mixtures were performed on the MIR-1K dataset [30] to verify the effectiveness of the proposed methods. This dataset consists of 1000 clips extracted from 110 karaoke recordings sung by 19 Chinese amateur singers. The sound files in this dataset contain instrument and vocal sounds in the left and right channels, respectively. All the stereo sound files are converted to mono and 16 kHz PCM format. We used clips of one male ('[PERSON]') and one female ('[PERSON]') singer for training set, and clips of the remaining 17 singers were assigned to the test set. Input mixture signals are generated by simple additions, \(y[n]=y_{1}[n]+y_{2}[n]\), for each pair of song clips. Figure 6 represents the distribution of \(\gamma_{1}\gamma_{2}\) on training set of MIR-1K dataset. \(\gamma_{1}\) is distributed highly around 0 and 1, which are from the regions where only a single source is present.  
The performance was evaluated by global normalized source to distortion ratio (GNSDR), global source to interference ratio (GSIR), and global source to artifact ratio (GSAR) values, which are the weighted mean of the NSDRs, SIRs, SARs, respectively, using each frame length as the weight of the corresponding frame [4; 26]. Normalized SDR (NSDR) is defined by an increase in SDR after separation. GSIR measures how much uninterested interference is present, GSAR measures how much irrelevant artifact sound exists in the separated results, and GNSDR considers both. The detailed definition of SIR, SDR, and SAR metrics can be found in BSS-EVAL 3.0 [26].  
Table 2 shows the results of the experiments performed on the MIR-1K dataset. Subscript \(m\) in \(GSIR_{m}\), \(GSAR_{m}\), and \(GNSDR_{m}\) means that the target source is musical instrument sound, so they measure the performances of instrument sound extraction from the input mixture. In the same way, \(GSIR_{v}\), \(GSAR_{v}\), and \(GNSDR_{v}\) measure the performances of vocal sound extraction from the mixture. For RNN, the three measures of instrument sound extraction, \(GSIR_{m}\), \(GSAR_{m}\), and \(GNSDR_{m}\), were all improved by the proposed curriculum learning. The interfering vocal sound was removed quite well as shown by the \(GSIR_{m}\) increment, from 11.72 dB to 12.30 dB. In the case of vocal sound extraction, \(GSAR_{v}\) was increased by 0.46 dB, but \(GSIR_{v}\) was decreased by 0.44 dB. It means that the proposed method left more music but less unwanted artifact in the separated vocal sound. The overall performance by \(GNSDR_{v}\) showed 0.27 dB increment with the proposed. For U-Net, the improvements were similar but mostly vocal extraction was better than music extraction. This might be caused by the property of U-Net using localized convolution windows that provide positive effects to vocal sounds in which gain and frequency characteristics change more often over time than instrument sounds.  
Figure 6: Distribution of \(\gamma_{1}\gamma_{2}\) on training set of MIR-1K dataset. \(\gamma_{1}\gamma_{2}\) is distributed between [0, 0.25].