# Target Speaker Extraction by Fusing Voiceprint Features
## 2 Methodology
### Overview of Tse-FV
#### 2.1.2 Feature Fusion  
The voiceprint features extracted from the reference speech are fused with the magnitude spectrograms extracted from the noisy audio using three different methods, i.e., Tse-FV(DC), Tse-FV(ECC), and Tse-FV(CEC). Finally, a set of fused features \(\mathbf{x}_{fuse}\) is constructed and fed into the next stage.  
Figure 1: The workflow of Tse-FV model. (**a**) Feature extraction. MFCCs \(\mathbf{r}_{fsc}\) and d-vectors \(\mathbf{r}_{dvec}\) are extracted from the reference speech. Magnitude spectrograms \(\mathbf{x}_{mag}\) are extracted from multispeaker signals. (**b**) Feature fusion. \(\mathbf{r}_{mfsc}\) and \(\mathbf{r}_{dvec}\) are concatenated in three ways, i.e., DC, ECC, and CEC, to construct the fused features \(\mathbf{x}_{fuse}\). (**c**) Speech separation. Three models (DPCL, PITNet, and VoiceFilter) are adopted to generate the estimated magnitude spectral mask corresponding to the target speaker, and the masked magnitude spectrograms are transformed to clean speech using the inverse STFT.  
Three proposed feature fusion methods are shown in Figure 1b. Specifically, the expanded convolutional neural network (CNN) is used to capture the low-level features of the audio signals. Because the expanded CNN was demonstrated to perform well in the task of audio feature extraction in [14], it was also adopted in our work. Denote the column number of the voiceprint features as \(D\), the number of time frames as \(T\), and the frequency as \(N\). Because the magnitude spectrogram feature \(\mathbf{x}_{mag}\in\mathbb{R}^{T\times N}\), and the d-vector \(\mathbf{r}_{dvec}\in\mathbb{R}^{1\times D}\), \(\mathbf{r}_{dvec}\) is copied \(T\) times along the first dimension (as shown in Equation (2)) to match the first dimension of \(\mathbf{x}_{mag}\). In the end, \(\mathbf{r^{\prime}}_{dvec}\in\mathbb{R}^{T\times D}\) is obtained and then concatenated to \(\mathbf{x}_{mag}\).  
\[\mathbf{r}_{dvec} =\begin{bmatrix}d_{1},&d_{2},&\dots&d_{D}\\ d_{1},&d_{2},&\dots&d_{D}\\ \vdots&\vdots&&\vdots\\ d_{1},&d_{2},&\dots&d_{D}\end{bmatrix}_{T\times D} \tag{2}\]  
**Direct Concatenation.** In the DC method, voiceprint features (\(\mathbf{r}_{mfc}\) or \(\mathbf{r^{\prime}}_{dvec}\), which are uniformly denoted by \(\mathbf{r}_{feat}\)) are directly concatenated to \(\mathbf{x}_{mag}\) along the first dimension using Equation (3).  
\[\mathbf{x}_{fuse}=\begin{bmatrix}\mathbf{x}_{mag};\mathbf{r}_{feat}\end{bmatrix} \tag{3}\]  
**Expanded Convolution and Concatenation.** In the ECC method, \(\mathbf{x}_{mag}\) is first input into the expanded CNN layer and then concatenated to \(\mathbf{r}_{feat}\) as  
\[\mathbf{x}_{conv} =\mathbf{CNN}(\mathbf{x}_{mag}) \tag{4}\] \[\mathbf{x}_{fuse} =\begin{bmatrix}\mathbf{x}_{conv};\mathbf{r}_{feat}\end{bmatrix}\]  
**Concatenation and Expanded Convolution.** In the CEC method, first \(\mathbf{r}_{feat}\) is concatenated to \(\mathbf{x}_{mag}\) and then the concatenated feature \(\mathbf{x}_{concat}\) is fed into the expanded CNN layer to obtain the fused feature \(\mathbf{x}_{fuse}\) using Equation (5).  
\[\mathbf{x}_{concat} =\begin{bmatrix}\mathbf{x}_{mag};\mathbf{r}_{feat}\end{bmatrix} \tag{5}\] \[\mathbf{x}_{fuse} =\mathbf{CNN}(\mathbf{x}_{concat})\]