# End-to-End Multi-Modal Speaker Change Detection with Pre-Trained Models
## 4 Experiments
### Results  
#### 4.6.1 Results of eMD Model  
Tables 3 and 4 detail the performance of the **eMD** model for SCD using freeze and non-freeze models across three modalities: Text, Audio, and Multi-modal.  
The Multi-modal approach consistently outperformed single modality configurations in both the freeze and non-freeze settings. For instance, in the freeze models, the Multi-modal modality achieved a Hit Rate of 79.88% in meeting EN2002a (45.51% P, 54.49% N), compared to 72.28% for Text and 78.34% for Audio. In the non-freeze models, the Multi-modal Hit Rate reached 82.98% in EN2002d (40.20% P, 59.80% N), surpassing Text (70.43%) and Audio (82.26%) modalities. The non-freeze models exhibited marked improvements over freeze models, particularly in Audio and Multi-modal settings. The FAR in Audio modality decreased from a range of 19.08-40.48% in freeze models to 8.32-25.99% in the non-freeze models. Similarly, the Multi-modal MDR reduced from 20.12-30.53% (freeze) to 17.02-33.33% (non-freeze), indicating enhanced detection accuracy.  
The class imbalance significantly impacted the Text modality in the freeze models, resulting in elevated MDR and reduced Hit Rates. For example, in meeting ES2004b (25.46% P, 74.54% N), the Text modality recorded an MDR of 43.72% and a Hit Rate of 56.28%, reflecting a bias toward the majority negative class. This imbalance compromised the detection of speaker changes. The non-freeze models, particularly Multi-modal, effectively addressed these challenges. In ES2004b, the Multi-modal non-freeze model improved the Hit Rate to 72.53% and reduced the MDR to 27.47%, demonstrating the advantage of adaptive learning and multi-modal integration in handling imbalanced distributions.  
Figure 2 compares the performance of the freeze (F) and non-freeze (NF) models across three setups of the **eMD** model: text-only, audio-only, and multi-modal. In Figure 2a, the stacked bar chart shows the average FAR and MDR. The hatched bars represent FAR, while the solid bars represent MDR. The light blue bars correspond to freeze mode, and the red bars correspond to non-freeze mode. Generally, the multi-modal configuration showsa lower FAR and MDR compared to the text-only and audio-only setups. All non-freeze models outperform the freeze models in all three settings. In the audio-only mode, it can be seen that the FAR decreases significantly from 31.49% to 25.98%, while the MDR drops from 25.24% to 16.93%.  
Figure 2b shows the averaged results of F1-scores. The text-only configuration achieves an F1-score of 54.22% with freeze and 55.26% with non-freeze. The audio-only configuration records an F1-score of 60.51% under freeze and 62.58% under non-freeze. The multi-modal configuration performs best, with an F1-score of 70.54% for freeze and 70.91% for non-freeze. The results demonstrate that the multi-modal approach outperforms both text-only and audio-only configurations, with a slight improvement in the non-freeze mode.  
\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline
**Meeting ID** & \multicolumn{3}{c}{**Text**} & \multicolumn{3}{c}{**Audio**} & \multicolumn{3}{c}{**Multi-Modal**} \\ \cline{2-11}  & **FAR** & **MDR** & **Hit** & **FAR** & **MDR** & **Hit** & **FAR** & **MDR** & **Hit** \\ \hline EN2002a & 39.79 & 29.57 & 70.43 & 25.99 & 18.69 & 81.31 & 25.04 & 17.25 & 82.75 \\ EN2002b & 34.66 & 33.2