# End-to-End Multi-Modal Speaker Change Detection with Pre-Trained Models
## 2 Related Work
During the training process, the pre-trained model is not trained; only the classification part is fine-tuned for low-latency SCD in broadcast news. The text-based and multi-modal-based SCD approach is another promising research direction [19; 20; 21; 22]. For the text-based approach, [PERSON] et al. [19], proposed a text-only speaker change detection method using word embeddings. Instead of relying on audio, the model uses fixed-length text windows and predicts speaker changes based on semantic similarity. The authors performed tests on Hebrew conversations, it achieved strong performance and generalized well to unseen speakers. The approach is useful in settings where audio is unavailable or not suitable. [PERSON] et al. [20] present a text-based SCD method that does not rely on audio signals. The authors propose a hierarchical recurrent neural network (RNN) architecture with static sentence-level attention, allowing the model to effectively incorporate surrounding context when determining speaker changes in text transcripts. The model processes dialogue in two levels: an LSTM encodes individual sentences, and another LSTM handles the sequence of sentence representations. A static attention mechanism helps the model focus on informative parts of the context. Experiments on a large dataset of TV talk show transcripts demonstrate that the method outperforms traditional and non-attention neural models, especially in settings where only textual data are available. For the multi-modal approach, in the work [21], the authors use the Transformer-Transducer (T-T) model to train a model to turn the audio into speaker turn augmented transcriptions, in which they added a special symbol in the transcriptions to indicate speaker turn for the next training process. Then, they used these transcriptions with a speaker turn symbol as training data and retrain with the previously obtained ASR model, then train a new model with a token-level loss function. [PERSON] et al. (2020) propose a hybrid approach to speaker change detection by combining both speech and text-based features. The authors use a Transformer-based model to detect speaker changes in ASR-generated transcripts, while also incorporating audio cues like speaker embeddings. By evaluating multiple fusion strategies (early, late, and hybrid fusion), the work demonstrates that joint modeling of audio and text outperforms using either modality alone. The method achieves strong results on broadcast news datasets, highlighting the complementary nature of acoustic and textual features in SCD. Another multi-modal approach was introduced by [PERSON] et al. (2019), a state-of-the-art multi-modal SCD model that combines both audio and text features using an encoder-decoder architecture. Instead of aligning speaker embeddings with individual words, the authors extract X-vector embeddings from 1.5 s audio windows, allowing for more robust speaker representation. The model architecture features modality-specific encoders (a BERT-based encoder for text and a CNN-based encoder for audio), followed by a Transformer decoder that performs multi-modal fusion and prediction. The system is trained using ASR-generated transcripts, not gold-standard text, making it suitable for real-world deployment. It outputs token-level speaker change predictions and is trained with a binary cross-entropy loss. An evaluation of the AMI dataset shows that the model outperforms previous single-modal and multi-modal baselines. The last two previous works are compared with the proposed approach in this study. The main difference of our model is the use of large pre-trained models for both text and audio modalities, along with the application of a self-attention layer to enhance audio feature extraction. To address the data imbalance issue, which is inherent to the speaker change detection task, we employ focal loss to further improve the model's performance.