# Pseudo-Phoneme Label Loss for Text-Independent Speaker Verification
## 3 Proposed Methods
### Pseudo-Phoneme Label Loss  
Similar to the HuBERT [38] and DeepCluster [39], we generate pseudo-phoneme labels by deep cluster, where the frame-level features before the pooling layer are clustered to obtain class centers, and the corresponding class that each frame feature belongs to is regarded as pseudo-phoneme labels. The loss calculation contains two parts: one is the clustering loss, and the other is the speaker loss. We randomly generate the same mean vector as the number of clusters, calculate the similarity between the frame-level features before the pooling layer and compute mean vector frame by frame to obtain the similarity matrix. The similarity matrix is compared with the pseudo-phoneme labels to obtain the clustering loss. When segment-level features are mapped to the speaker category number, we will compute the CE loss by using the ground truth speaker labels. Finally, the two are summed up as the total loss of the model. The overall architecture of the network is shown in Figure 1, and the process of loss calculation is shown in Figure 2.  
The cluster loss is calculated as follows: we feed the deep neural network with acoustic features, and then the output before the pooling layer is noted as the frame-level feature \(\mathbf{X}\), \(\mathbf{X}=[x_{1},x_{2},\cdots,x_{l}]\), where \(l\) is the number of feature frames, and the feature dimension of each frame is \(D\). \(Original\) represents the \(\mathrm{D}\times\mathrm{L}\) dimensional feature vectors output from the TDNN layer. We use unsupervised clustering methods to obtain the pseudo labels by setting the number of categories, performing the deep cluster method on the frame-level features to obtain the corresponding categories and computing \(\mathcal{L}_{PPL}\) by combining the similarity matrix \(M\) with the pseudo-label \(L_{pseudo}\), where the number of corresponding  
Figure 1: The x-vector architecture for speaker recognition. Using joint learning, the frame-level information from the TDNN layer is compared with the center vector in the PPL layer to calculate the PPL loss, which is then summed with the CE loss to obtain the total loss. Figure 2: The computation process of PPL loss. \(Original\) represents the \(\mathrm{D}\times\mathrm{L}\) dimensional feature vectors output from the TDNN layer, and \(PPL\)\(layer\) represents the \(\mathrm{D}\times\mathrm{C}\) dimensional center vectors maintained by the learnable dictionary encoding (LDE) layer. The similarity matrix \(\mathbf{M}\) is calculated by the feature of the TDNN layer and the transpose feature of the LDE layer, obtaining a result of a \(\mathrm{L}\times\mathrm{C}\) matrix. In the matrix \(\mathbf{M}\), each row represents the similarity between each frame feature of the input and the C center vectors in the PPL layer, and each color represents a class. After that, the corresponding class of each frame feature is the pseudo-phoneme label \(L_{pseudo}\) according to the clustering result. Finally, the CE loss is calculated between M and \(L_{pseudo}\) to obtain the PPL loss. categories is the pseudo-label \(L_{pseudo}\). Each mean vector is taken \(\mathbf{\mu}\) as a cluster center, where \(\mathbf{\mu}=[\mu_{1},\mu_{2},\cdots,\mu_{c}]\), \(c\) is the number of class centers and \(D\) is the dimension. Subsequently, we calculate the distance between frame-by-frame features and class centers to get the similarity matrix \(\mathbf{M}\), with dimension \(t*D\), in order to finish the loss calculation. In order to get the final loss, we average the classification loss and PPL loss and then add them together.