# Target Speaker Extraction by Fusing Voiceprint Features
## 2 Methodology
### Overview of Tse-FV
#### 2.1.3 Target Speech Extraction  
As shown in Figure 1c, the speech separation models predict the masked magnitude spectrograms corresponding to the target speaker based on the fused features \(\mathbf{x}_{fuse}\). Three speech separation models were explored: (1) the DPCL model [6; 7] consisting of four LSTM layers, a dropout layer, and a linear layer; (2) the PITNet model [9] consisting of four LSTM layers, a dropout layer, and two linear layers; and (3) the VoiceFilter [14] model consisting of three LSTM layers and two linear layers. Both DPCL and PITNet are speaker-independent speech separation models which output several estimated spectrograms according to a preset number of speakers without prior knowledge of speakers. In order to apply the speaker's voiceprint features into DPCL and PITNet, a new feature integration method is proposed as shown below. First, DPCL and PITNet adopt LSTM to generate an embedding \(\mathbf{e}\) for each T-F bin in \(\mathbf{x}_{fuse}\) as  
\[\mathbf{e}=\text{LSTM}(\mathbf{x}_{fuse};\mathbf{\theta}) \tag{6}\]  
where \(\mathbf{\theta}\) is a model parameter vector. The speaker-dependent voiceprint information contained in \(\mathbf{x}_{fuse}\) helps to make embeddings of the target speaker's T-F bins more distinctive from the others. Then, DPCL uses the K-means clustering algorithm to generate a partition of the embeddings \(\mathbf{m}_{1},\cdots,\mathbf{m}_{C}\) corresponding to a preset number of speakers \(C\) as  
\[\mathbf{m}_{1},\cdots,\mathbf{m}_{C}=\text{K-Means}(\mathbf{e},C) \tag{7}\]  
where \(\mathbf{m}_{1},\cdots,\mathbf{m}_{C}\) are called the magnitude spectrogram masks. By contrast, PITNet uses linear layers to estimate \(\mathbf{m}_{i}\) for all embeddings as  
\[\mathbf{m}_{1},\cdots,\mathbf{m}_{C}=g_{linear}(\mathbf{e};\mathbf{\theta});\mathbf{m}_{i}\geq 0, \sum_{i=1}^{C}\mathbf{m}_{i}=1 \tag{8}\]  
To extract the target speaker's speech, DPCL and PITNet are designed to output two spectrogram masks, i.e., \(C\) equals 2. One spectrogram mask corresponds to the target speaker and the other corresponds to all the other speakers including the background noise. However, which spectrogram mask corresponds to the target speaker has not been determined yet. Then, \(\mathbf{m}_{i}\) is applied on \(\mathbf{x}_{mag}\) to generate the \(i\)th estimated magnitude spectrograms \(\hat{\mathbf{x}}_{mag}^{i}\) for the \(i\)th speaker using Equation (9), where \(\odot\) denotes the Hadamard Product. \[\hat{\mathbf{x}}_{mag}^{i}=\mathbf{m}_{i}\odot\mathbf{x}_{mag} \tag{9}\]  
After that, \(\hat{\mathbf{x}}_{mag}^{i}\) is transformed into waveforms \(\hat{\mathbf{y}}_{i}\) using the inverse STFT. The cosine similarity between the referenced d-vector \(\mathbf{r}_{dvec}\) and the d-vector of the generated speech \(\hat{\mathbf{y}}_{i}\), i.e., \(\mathbf{r}_{dvec}^{i}\), are computed and the speech \(\hat{\mathbf{y}}\) corresponding to the largest similarity is regarded as the target speaker's speech as shown in Equation (10). \[\hat{\mathbf{y}}=\operatorname*{arg\,max}_{\hat{\mathbf{y}}_{i}}\cos(\mathbf{r}_{dvec}, \mathbf{r}_{dvec}^{i}) \tag{10}\]  
Different from DPCL/PITNet, VoiceFilter is a speaker-dependent model which only outputs the estimated magnitude spectrogram mask corresponding to the target speaker. It adopts LSTM to predict the target magnitude spectrogram mask \(\mathbf{m}\) based on the inputs of \(\mathbf{r}_{dvec}\) and \(\mathbf{x}_{mag}\). In this work, \(\mathbf{r}_{dvec}\) was substituted by \(\mathbf{r}_{mfc}\) and \(\mathbf{m}\) was obtained using the speech separation module of VoiceFilter based on \(\mathbf{x}_{fuse}\). After obtaining the magnitude spectrogram mask \(\mathbf{m}\) corresponding to the target speaker, the estimated magnitude spectrogram feature \(\hat{\mathbf{x}}_{mag}\) corresponding to the target speaker is generated by multiplying \(\mathbf{m}\) with \(\mathbf{x}_{mag}\) using Equation (9).