# End-to-End Multi-Modal Speaker Change Detection with Pre-Trained Models
## 1 Introduction  
Speaker change detection (SCD) is an important task in audio processing, particularly for applications such as speech segmentation, speaker diarization, and automatic transcription. Accurate detection of speaker transitions is critical in multi-person conversations, especially in meeting scenarios, where challenges such as overlapping speech and background noise can make correct detection difficult.  
Traditional SCD methods rely on audio features, including pitch, Mel Frequency Cepstral Coefficient (MFCC), and cepstral coefficients, with various approaches such as metric-based and neural network-based methods. However, these approaches often underperform in complex acoustic environments such as those with low signal-to-noise ratios or simultaneous speech. Other multi-modal approaches to SCD have been explored, but they face practical limitations. Their approaches rely on ground-truth transcripts, which are often impractical in real-world scenarios due to transcription errors and limited availability. Another issue is that binary cross-entropy loss (BCE loss) is used as the main objective function, which causes models to fail to learn useful features from imbalanceddata. Usually, an SCD dataset contains many negative samples (no change points) and fewer positive samples (change points); it has imbalanced features.  
To address these challenges, we propose an end-to-end multi-modal SCD model with a self-attention layer based on large pre-trained models and enhance it with focal loss to mitigate the issue of imbalanced data. The model employs a pre-trained large audio model to extract acoustic features from raw audio and a large language model (LLM) to derive semantic and contextual information from transcripts generated by an automatic speech recognition (ASR) system. By leveraging both modalities in a practical, real-world setting, this approach enhances detection accuracy in challenging acoustic conditions while avoiding the reliance on manually annotated transcripts. Furthermore, this study investigates the impact of different configurations, including audio-only, text-only, and multi-modal setups, to evaluate their effectiveness in SCD. A comparison between BCE loss and different focal loss configurations is carried out. Additionally, we explore different fine-tuning strategies, examining the impact of updating the pre-trained models to optimize performance while balancing computational efficiency. The evaluation is conducted using the multi-talker meeting corpus [1], a widely recognized dataset in speech and audio processing research. Experimental results showed that updating the pre-trained model's parameters during the training significantly outperformed freezing them for all modalities. Compared to the text-only model, the audio-only model outperforms it in various settings, and the multi-modal configuration surpasses both. The self-attention layer introduced in the approach effectively improved the F1-score compared to the model without using it for various modalities. The proposed approach outperforms the baselines and also outperforms previous approaches enhanced with multi-modals.  
The rest of the work is organized as follows: (i) Section 2 reviews related studies on SCD, including multi-modal approaches. (ii) Section 3 presents the approach, including task formalization and the proposed model. (iii) Section 4 provides details on the experiments, including model setup, data statistics, baselines, and a discussion of the results. (iv) Section 5 describes a discussion related to the obtained results. (v) Section 6 discusses the limitations of the approach and potential future directions. (vi) Section 7 concludes the work with a summary of the key findings.