# End-to-End Multi-Modal Speaker Change Detection with Pre-Trained Models
## 2 Related Work  
Existing approaches to SCD problems can be categorized into two groups: (i) unsupervised approaches; (ii) supervised approaches. Most unsupervised approaches are considered metric-based methods that calculate the discrepancy between two consecutive frames or segments. A change point is detected if the distance between the two segments exceeds a predefined threshold; otherwise, it is considered that no change point exists. In this direction, one of the simplest metric-based approaches [2] is to use a distance function, such as Kullback-Leibler (KL) divergence [3], to measure the similarity between two consecutive segments modeled with Gaussian mixture models (GMMs). It is very sensitive to a predefined threshold and difficult to generalize to unseen audio data. Early methods to SCD are the combination of GMMs with Bayesian information criterion (BIC) [4; 5; 6]. Specifically, each segment will be modeled with separate GMMs (referred to as two speaker models), and two consecutive segments will be modeled with a GMM (referred to as a single speaker model). BIC is used to calculate a score that estimates how well these GMMs will predict the given segments. Generalized Likelihood Ratio (GLR) [7] can also be applied with GMMs. It measures the likelihood ratio between two distributions of these segments. Studies have shown that using neural network-based speaker embeddings [8; 9] outperforms traditional audio features such as mel-frequency cepstral coefficients (MFCCs), pitch, and filter banks in SCD. In the work [10],the authors applied a neural network to calculate speaker embeddings at the frame level, such as d-vector, and used it with metric-based segmentation to detect speaker changes. It achieved a fast and effective SCD model using deep speaker vectors. In the supervised direction, many machine learning methods have been applied to SCD, including conditional random field (CRF) [11], which was applied to speaker detection and tracking. In their work, the authors proposed a variational CRF training algorithm for SCD, and it outperforms GMM-based methods; however, it requires knowing the speaker number in the audio in advance. In addition to traditional models, various neural networks [12; 13] have been applied to SCD in recent years. In the work [13], the author proposed an end-to-end SCD approach which also covers overlap speaker change issues. They consider this as a multi-label classification task, which is performed with permutation-invariant training. SincNet [14] and bidirectional Long Short-Term Memory (LSTM) were applied on short audio chunks with higher temporal resolution. The authors reported that their results are better on both voice activity detection and audio segmentation. Beyond SCD-specific methods, several studies have contributed important insights into improving ASR systems more generally. In a practical application, [PERSON] et al. [15] developed a multi-modal ASR system for speech therapy that integrates audio and visual cues, demonstrating improved recognition in noisy environments. [PERSON] et al. [16] showed that language models trained on speech transcripts significantly outperform those trained on written texts, even when using smaller corpora. [PERSON] and [PERSON] [17] highlighted how architectural choices in phoneme recognition systems affect performance, especially when combining modular components. To address noise robustness, [PERSON] [18] introduced a Bayesian framework that unifies front-end and back-end compensation techniques. In recent developments, studies on SCD based on pre-training technologies have become more frequent. In the work [12], the authors proposed a model for speaker embeddings, which was pre-trained on three conditions: (i) gender classification; (ii) contrastive loss; (iii) triplet loss. Then, the model takes two audio segments (2 s) and feeds them into a classification layer to identify whether they belong to a speaker or more.