# Improving Singing Voice Separation Using Curriculum Learning on Recurrent Neural Networks
## 3 Proposed Curriculum Learning Technique on RNN and U-Net
### Proposed Curriculum Learning for Singing Voice Separation  
Stochastic models can learn more effectively by dividing the training phase and increasing the degree of difficulty in sequence according to the phase, as known as curriculum learning [11]. Examples of applying curriculum learning to audio data include speech emotion recognition [27], and speech separation [28]. However, the proposed method differs in that the difficulty is determined using the source dominance of each time-frequency bin. In order to apply curriculum learning to singing voice separation, the difficulty level of each data sample is defined. Our main assumption is that if one source signal is dominant to the other, it is more effective in describing the corresponding source. The dominance of source 1 is defined by the ratio of source 1 to the sum of source 1 and 2, in terms of their powerspectral energies as follows:  
\[\gamma_{1}(t,f)=\frac{\left|y_{1,t}(f)\right|^{2}}{\left|y_{1,t}(f)\right|^{2} +\left|y_{2,t}(f)\right|^{2}}\,, \tag{13}\]  
and likewise, the dominance of source 2 is computed as  
\[\gamma_{2}(t,f)=\frac{\left|y_{2,t}(f)\right|^{2}}{\left|y_{1,t}(f)\right|^{2} +\left|y_{2,t}(f)\right|^{2}}\,. \tag{14}\]  
The relationship between the dominance factors of source 1 and source 2 is that \(\gamma_{2}=1-\gamma_{1}\). The multiplication of the two dominance factors, \(\gamma_{1}\gamma_{2}=\gamma_{1}(1-\gamma_{2})\), is considered to model the reciprocal effect of the dominances of the two exclusive sources. Figure 4 shows the value of \(\gamma_{1}\gamma_{2}\) according to the change of \(\gamma_{1}\), the dominance of a single source. Same behavior is observed for \(\gamma_{2}\) as well. It shows that if only a single source is active, \(\gamma_{1}=1\) or \(\gamma_{2}=1\), \(\gamma_{1}\gamma_{2}\) becomes zero, which is the minimum, because the other dominance factor is zero. In that case, no separation processes are required, so we can regard this obviously the easiest case. If both sources are active by the same degree, \(\gamma_{1}=\gamma_{2}=0.5\), \(\gamma_{1}\gamma_{2}\) is maximum (0.25), and it is the most difficult case. To selectively give weights according to the difficulty of the separation, we propose the following weight function:  
\[w(t,f)=1+\alpha\gamma_{1}(t,f)\gamma_{2}(t,f)\,,\quad\alpha\in\left\{-1,0,1 \right\}, \tag{15}\]where \(\alpha\) is a mode selection parameter that may vary according to the curriculum. As shown in Figure 4, \(\gamma_{1}\gamma_{2}\) is limited to \([0,0.25]\), so \(w(t,f)\) is limited to \([1,1+0.25\alpha]\). Figure 5 shows \(w(t,f)\) with different \(\alpha\) values assigned. If \(\alpha=1\), \(w(t,f)\) grows as \(\gamma_{1}\) increases, and has the maximum at \(\gamma_{1}=0.5\), which is the most difficult case. For \(\alpha=-1\), the graph is upside down, so the easiest cases (\(\gamma=0\) and \(1\)) will have the maximum weights. We apply the weight in Equation (15) to the loss function in Equation (4) to obtain a new weighted loss function:  
\[J_{w}=\frac{1}{TF}\sum_{t=1}^{T}\sum_{f=1}^{F}w(t,f)\left[\left(y_{1,t}(f)- \tilde{y}_{1,t}(f)\right)^{2}+\left(y_{2,t}(f)-\tilde{y}_{2,t}(f)\right)^{2} \right]\,. \tag{16}\]  
Figure 4: \(\gamma_{1}\gamma_{2}\) value with respect to \(\gamma_{1}\). This value is close to zero when one source is dominant, and has a maximum value of \(0.25\) when two sources are evenly mixed. Figure 5: Loss weight \(w(t,f)\) with respect to \(\gamma_{1}\) when \(\alpha=1\) (**solid line**) and \(\alpha=-1\) (**dashed**). In the case of \(\alpha=1\), time-frequency bins which two sources are mixed equally has large weights. In the case of \(\alpha=-1\), ones which one source is dominant has large weights. In Equation (15), we can use \(\alpha\) to choose where to focus during learning. The value of \(\alpha\) is set differently according to the learning stages to obtain the proposed curriculum learning algorithm. Here, we can choose \(\alpha\in\{-1,0,1\}\).