# Improving Singing Voice Separation Using Curriculum Learning on Recurrent Neural Networks
## 3 Proposed Curriculum Learning Technique on RNN and U-Net
### U-Net Baseline Model  
Our U-Net baseline model is based on Reference [7]. For the mixed input signals, STFT extracts magnitude spectrogram \(\mathbf{X}\) of size \(T\times F\), where \(T\) is the number of frames in time domain and \(F\) is the number of frequency bins. Each component of the 2-dimensional matrix \(\mathbf{X}\) is a specific time-frequency unit region,  
\[\mathbf{X}(t,f)=x_{t}(f)\,, \tag{9}\]  
Figure 3: Stacked RNN baseline architecture. Baseline model consists of three layers of multiple RNNs and a fully connected (FC) layer. Individual sources have their own FC layer, and the weight and bias parameters of RNN layers are shared. where \(t\) and \(f\) are discrete time and frequency index variables, and \(x_{t}(f)\) is STFT of the input mixture sound at time \(t\) and frequency \(f\). The spectrogram matrix \(\mathbf{X}\) is globally scaled so that all the values of \(\mathbf{X}\) should be within \([0,1]\) for fast and stable learning. The encoder consists of six convolution layers with various kernel window sizes and stride values. Each convolution output is generated according to the following formula:  
\[\mathbf{E}^{(l)}=\left\{ \begin{array}{ll}\mathbf{X}\,,&l=0\\ g\left(\text{conv}\left(\mathbf{E}^{(l-1)},\mathbf{W}_{e}^{(l)},\mathbf{b}_{e }^{(l)}\right)\right)\,,&l\in\{1,\ldots,6\}\,,\end{array} \right. \tag{10}\]  
where \(g\) is an activation function, \(l\) is the layer number expressed by parenthesized superscripts, \(\mathbf{W}_{e}^{(l)}\) is a matrix whose column vectors are convolution kernel functions, and \(\mathbf{b}_{e}^{(l)}\) is a bias vector. The superscript and the subscript notations, \(\{\cdot\}_{e}^{(l)}\), indicate that the parameter set belongs to layer \(l\) of the encoder network. The initial output is a copy of the input spectrogram, \(\mathbf{E}^{(0)}=\mathbf{X}\). The function \"conv\" defines a 2-dimensional convolution operator with the given set of a kernel matrix and a bias vector. The encoder extracts high-level features with resolution reduction from the input features. Decoding part of the U-Net consists of five up-convolution layers with initial input as the encoder output at the final convolution layer. The outputs of the encoder layers are concatenated to the inputs of the decoding layers to compensate any lost information during encoding. The decoding operation from lower layer to the upper layer is recursively defined as follows:  
\[\mathbf{H}^{(l)} = \left\{ \begin{array}{ll}\mathbf{E}^{(l+1)}\,,&l=5\\ \left[\mathbf{E}^{(l+1)},\mathbf{D}^{(l+1)}\right]\,,&l\in\{0,\ldots,4\}\,, \end{array} \right.\] \[\mathbf{D}^{(l)} = g\left(\text{upconv}\left(\mathbf{H}^{(l)},\mathbf{W}_{d}^{(l)},\mathbf{b}_{d}^{(l)}\right)\right)\,, \tag{11}\]  
where \([\cdot]\) is a concatenation operator, \(\mathbf{W}_{d}^{(l)}\) and \(\mathbf{b}_{d}^{(l)}\) are a matrix of the deconvolution kernel functions and a bias vector, and \(\mathbf{H}^{(l)}\) is an intermediate output at layer \(l\). The initial input to the decoding up-convolution layer is the encoder output only, that is, \(\mathbf{H}^{(5)}=\mathbf{E}^{(6)}\). The detailed configuration including the sizes and the numbers of the kernel functions adopted in this paper is given in Table 1. Learning the U-Net baseline is briefly summarized in Algorithm 2. Exact calculation of the gradients can be found in Reference [7].