# Speaker Diarization: A Review of Objectives and Methods
## 7 Clustering Frames by Speaker Turns  
In sections of speech, as perhaps estimated by SAD, SD must determine (1) whether one or multiple speakers are talking and (2) at what times speakers start and cease talking. One common approach is to seek times of pertinent change in speech features and/or group (cluster) related speech frames into coherent speaker segments [43; 58; 59; 60; 61]. Section 2 described features of speech that may be exploited to seek speech continuity. However, given the complexity of speech, most approaches to SD have used more general distance measures. Other earlier methods also used HMM-ASR to assist [58]. ASR can assist in modern neural SD as well [62], as ASR can provide textual clues to speaker turns.  
To measure the distance between speech segments, there are two common methods for SD: _generalized likelihood ratio_ (GLR) [63] and _Bayesian information criterion_ (BIC) [64]. BIC searches for change points within a time window, using a penalized likelihood ratio test of whether data are better modelled by a single or different distributions. GLR is a likelihood-based metric: the ratio between two hypotheses (whether successive segments come from the same speaker or not). Also common is the _Kullback-Leibler_ divergence, which estimates the distance between two random distributions [65].  
Other popular techniques include _spectral clustering_ (SC) [51], _k-means clustering_[66; 67; 68], _mean shift clustering_[56], and _agglomerative hierarchical clustering_ (AHC) [69] to aggregate the regions of each speaker into separate clusters. These tend to use a bottom-up approach that trains multiple clusters or models and successively merges them, reducing the number of clusters until one remains per speaker. SC mostly uses a cosine distance, rather than the Euclidean.  
However, many clustering methods can only estimate one speaker per segment and thus cannot handle speech of overlapping speakers. Also, standard k-means clustering uses squared error as a criterion, which generally does not handle non-Gaussian data well nor data containing imbalanced classes, which are common for speech. Indeed, many SD change-point detections are based on metric-based approaches that assume the distribution of speech features follow a Gaussian distribution, which is often not the case.  
SC is a common unsupervised clustering technique based on graph theory [70]. It requires a pairwise similarity score between all pairs of speaker embeddings, resulting in a similarity graph. SC is sensitive to noise in the affinity matrix.  
AHC constructs a hierarchy based on distances between features and forms a group. AHC is a common unsupervised clustering technique, which uses pairwise distances between all speaker embeddings, yielding a distance matrix. A cluster hierarchy is generated (a tree diagram called a dendrogram). Each input sample starts in its own cluster, and pairs of clusters are then merged.  
Variations on clustering include adding constraints [71,72] and affinity propagation [73], as well as X-means and online re-clustering [66,74]. There is also Viterbi realignment, which re-segments an audio stream based on a current clustering hypothesis, with re-training on an alternative segmentation [75,76]. Most clustering in SD is unsupervised [77]. Some SD studies have used mean-shift clustering, which assigns data points to clusters iteratively via modes in a non-parametric distribution [78].  
In _deep embedded clustering_ (DEC) [79], input features (speaker embeddings) are transformed to be more separable. For differentiability (needed to train ANNs), each embedding has a probability for available speaker clusters, which are iteratively refined using bottleneck features estimated by an autoencoder [80]. Some SD approaches combine segmentation and clustering into one trainable model [77].