# Improving Singing Voice Separation Using Curriculum Learning on Recurrent Neural Networks
## 2 Related Work
### RNN-Based Singing Voice Separation  
RNN is a network created to process sequential data using memory, in other words, hidden states that are invisible to the outside of the network. A vanilla RNN conveying basic elements only is defined by the following equation:  
\[\mathbf{h}_{t}=g\left(\mathbf{W}_{ht}\mathbf{h}_{t-1}+\mathbf{W}_{hx}\mathbf{ x}_{t}+\mathbf{b}\right), \tag{1}\]  
where \(t\) is a discrete time index, \(\mathbf{h}_{t}\) is a hidden state output vector at time \(t\), \(\mathbf{x}_{t}\) is an input vector whose components are magnitude spectra generated by STFT, \(\mathbf{b}\) is a bias vector, and \(g\) is an activation function. The input, state variable, and bias variable represented by boldface letters are all vectors. \(\mathbf{W}_{ht}\) is a weight matrix from the past output (\(\mathbf{h}_{t-1}\)) to the current output vectors (\(\mathbf{h}_{t}\)), and \(\mathbf{W}_{hx}\) is another weight matrix from the input feature (\(\mathbf{x}\)) to the output. RNN can process sequences of arbitrary length in such a way that hidden state \(\mathbf{h}_{t-1}\) summarizes the information of the previous inputs, \(\{\mathbf{x}_{1},\ \ldots,\mathbf{x}_{t-1}\}\), and combines them with the current input \(\mathbf{x}_{t}\) to calculate the output vector \(\mathbf{h}_{t}\). According to the first Markovian assumption in Equation (1), the vanilla RNN assumes dependency to previous output only, so it may not be able to handle the cases where the dependency is complicated and exists over long time. To model multiple-level dependency in time, several vanilla RNNs are connected sequentially to  
Figure 1: Illustration of a typical deep neural network framework for singing voice separation. The input and output features are the magnitude spectra of the short-time Fourier transform (STFT) of the time-domain signals. Neural network models are applied to separate vocal and instrument sources in the magnitude STFT domain, and the audio sounds are reconstructed by inverse STFT operations.  
build stacked RNNs as shown in Figure 2. Stacked RNN consists of several layers of multiple RNNs, and longer time dependencies are expected to be modeled by the cascaded recurrent paths.  
The dependency across both time and frequency axes can be modeled by convolutional neural networks (CNNs) [19; 20]. The CNN architecture is not well suited to sequential data such as audio sounds because the size of its receptive field is fixed, and the convolution operation calculates the output based only on the region within the receptive field. However, CNN has its own advantage that most calculations can be implemented in a parallel manner, which, in terms of computational efficiency, makes a great advantage over RNN that requires computation of all previous outputs to be finalized before generating the current outputs. For these reasons, CNN and its variants [1; 2; 3] have been widely used for singing voice separation. In addition, there have been attempts to combine an RNN and a CNN. One of their combinations is a convolutional recurrent neural network (CRNN) that has been successfully adopted to singing voice separation as well [17; 18].