# A Multi-Source Separation Approach Based on DOA Cue and DNN
## 3 Multiple Sound Source Separation Framework
### Non-Sparse Component Point Recovery  
In Section 3.1, we can obtain the separated components from the sparse component points. However, if we only focus on sparse component point recovery and ignore the non-sparse component point, the separated signals lose a lot of TF components. In particular, the number of non-sparse component points in the recorded signal increases with the increase in the number of sound sources. If only sparse component point recovery is performed, it is difficult to acquire a good perceptual quality of the separated signal. Hence, the non-sparse component point recovery is very important for improving the quality of the separated speech. Therefore, in this subsection, the recovery method for the non-sparse component point is introduced.  
The non-sparse component points are composed of the superposition of the components of multiple sound sources and/or reflection components. Since reflection components in the recorded signal have been mostly removed by the method introduced in Section 2, the non-sparse component points are considered to be mainly composed of multiple sound source components in this subsection. Therefore, the non-sparse component point recovery problem can be transformed to calculate the proportion of each sound source at each TF point, which can be denoted by an ideal ratio mask (IRM).  
In recent years, deep neural networks (DNNs) have achieved excellent results in many fields, and they are very suitable for solving the regression problem. In addition, calculating the IRM for each TF point can be regarded as a regression problem. Therefore, a DNN is employed to predict the IRM and to achieve non-sparse component point recovery. The amplitude of the TF domain coefficient of one frame mixed signal is input into the DNN. The training target of the network is the IRM for each sound source, which can be formulated as follows:  
\[\text{IRM}(n,k)=\frac{|S_{i}(n,k)|^{2}}{\sum_{i=1}^{N}|S_{i}(n,k)|^{2}} \tag{22}\]  
where \(S_{i}(n,k)\) is the TF domain coefficient of the \(i_{th}\) sound source; \"\(|\cdot|\)\" represents the amplitude of a complex, and \(N\) is the number of sound sources.  
The neural network structure is shown in Figure 7. The neural network consists of two full connection layers and two convolution layers. The first full connection layer and the two convolutional layers are followed by a Relu active function, a batch-normalization layer, and a dropout layer. The second full connection layer is followed by a sigmoid layer.  
In addition, we employ the Adam optimizer to train the neural network. The network is trained based on the minimum root mean square error (RMSE) criterion. The RMSE cost function \(\mathcal{F}\) can be expressed as:  
\[\mathcal{F}(\text{IRM}(n,:);\mathbf{\Theta})=\sqrt{\frac{1}{M}\sum_{k=1}^{M}\left[ IRM(n,k)-\mathcal{L}(\mathbf{g}(n))\right]^{2}} \tag{23}\]  
where \(M\) is the frequency channel number, \(\mathbf{\Theta}\) is the parameters of DNN, \(\mathcal{L}(\cdot)\) represents prediction operation of DNN, and \(\mathbf{g}\) is the input feature vector (i.e., the magnitudes of TF representation of total TF points in one frame).  
In the test phase, the trained network with \(\mathbf{\Theta}\) minimizing RMSE can be used to predict the IRM, namely, the proportion of the energy of each sound source at each TF point.