# U-Vectors: Generating Clusterable Speaker Embedding from Unlabeled Data
## 3 Methodology
### Problem Formulation and Assumptions  
The proposed method tends to solve the speaker recognition system in an unsupervised manner based on some constraints. Table 1 summarizes the paper's mathematical notations to facilitate the readers. To comprehend the problem statement, let \(\mathcal{S}\) be a database of speech segment, where \(\mathcal{X}_{k}\) be a short-length audio segment containing speech of an individual. Also, let \(x_{i}\) be a smaller window/frame of the audio segment, where \(x_{i}\in\mathcal{X}_{k}\). From a particular speech segment, \(\mathcal{X}_{k}\), \(\mathcal{M}\) number of non-overlapping speech frames are generated. As it is stated that a speech segment belongs to a single individual, the smaller speech frames also belong to that individual. From this intuition, we construct pairwise constraints between audio frames. We define two speech frames belong to the same cluster if they belong to the same audio segment. On the contrary, we consider two speech frames that belong to different clusters if they belong to different audio segments. Based on the pairwise relations, a set of cluster \(\mathcal{C}\) can be generated. Where a single cluster (\(c_{i}\in\mathcal{C}\)) belongs to a specific speech segment. Considering most speech segments will belong to a single speaker, we can assume that most cluster \(c_{i}\) would contain a single individual's data. However, as multiple speech segments can belong to a single individual, multiple clusters may contain a single individual's data. Hence, the challenge is to find such optimal cluster relationships such that no two clusters may contain speech of a single individual. The training strategy has two possibilities, such as:  
* In the case of an unlabeled dataset, let us consider that each audio file contains a single individual's speech. For a set of audio files with an unknown number of speakers, our approach is suitable to produce clusterable embeddings based on speakers. This constraint is similar to semi-supervised learning, as some of the pairwise constraints are known [33]. * Let us consider a dataset containing multiple speakers' conversations, where a single audio stream may include various speakers. In such a case, no pairwise constraints are known, and an unsupervised strategy is required. Hence, we produce hypothetical pairwise constraints based on audio segmentation processes such as VAD, word segmentation [5], etc., and construct pairwise constraints. However, in such a case, the embedding system's accuracy depends on the purity of the audio segmentation process. Figure 2: The figure illustrates the comprehensive procedure to generate u-vectors. Audio stream is segmented and given pseudo labels. Then a cluster network is formed using small non-overlapping speech frames from the speech segments. Finally, training data is generated from the cluster network based on the needs of the siamese architecture. The pairwise network requires an equal number of can-link and cannot link pairs. In contrast, triplet networks receive three data, one pair with can-link, and one cannot-link pair. The network/embedder is trained on randomly augmented speech frames, further converted into 2D feature maps using MFCC. While training, the embedder reforms the cluster associations, where speaker similarity is considered. For both of the problem, DL architecture is used to aggregate multiple clusters such that the resulting cluster contains all of the embeddings of a single individual. We imply that if a DL function can properly extract speech features from audio frames, it can obtain optimal reasoning of speech frames being similar and dissimilar. Further, an optimally trained DL framework can successfully re-cluster the data based on the feature similarity rather than the number of hypothetical clusters. We extend our investigation towards finding such DL training strategy.