# End-to-End Multi-Modal Speaker Change Detection with Pre-Trained Models
## 4 Experiments
### Results
#### 4.6.4 Comparison with Baselines  
The performance of the **eMD** and **eMD\({}_{att}\)** models is evaluated in Table 7, which indicates improvements of up to 2.73% in the F1-score over the baseline Wav2 vec-MLP. The comparison includes baseline models: Neural Network (NN), Long Short-Term Memory (LSTM), and Wav2 vec-MLP. The NN struggles with a high missed detection rate, while LSTM shows improvement but remains weak in detecting speaker changes. Wav2 vec-MLP provides a stronger baseline with higher precision, though it still misses some detections.  
Figure 3: Comparison of freeze (F) and non-freeze (NF) models across text-only, audio-only, and multimodal configurations of **eMD\({}_{att}\)** model. (**a**) Average FAR and MDR (%) Comparison: F vs. NF. (**b**) Average F1-score (%) Comparison: F vs. NF.  
\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Models** & **FAR** & **MDR** & **Precision** & **Recall** & **F1** \\ \hline NN & 7.66 & 71.11 & 62.74 & 28.89 & 38.16 \\ \hline LSTM & 10.21 & 56.65 & 65.97 & 43.35 & 50.99 \\ \hline Wav2 vec2-MLP & 8.25 & 35.03 & 78.47 & 64.97 & 70.66 \\ \hline
**eMD** (audio-only) & 16.93 & 25.24 & 67.06 & 74.76 & 70.54 \\ \hline
**eMD** (multi-modal) & 17.07 & 24.43 & 67.02 & 75.57 & 70.91 \\ \hline
**eMD\({}_{att}\)** (audio-only) & 25.69 & 10.92 & 61.14 & 89.08 & 72.39 \\ \hline
**eMD\({}_{att}\)** (multi-modal) & 21.78 & 14.15 & 64.30 & 85.85 & 73.39 \\ \hline
**eMD\({}_{att}^{*}\)** (multi-modal) & **15.88** & 20.77 & **69.61** & **79.23** & **74.01** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Averaged results of **eMD**, **eMD\({}_{att}\)**, and **eMD\({}_{att}^{*}\)** compared with baselines. The asterisk (*) denotes the model optimized with focal loss parameters. Bold values indicate the best performance among all models for each metric.  
Figure 4: Performance metrics (F1-score, False Alarm Rate, and Miss Detection Rate) as a function of \(\alpha\) for different \(\gamma\) values: (**a**) False Alarm Rate as a function of \(\alpha\) for different \(\gamma\) values. (**b**) Miss Detection Rate as a function of \(\alpha\) for different \(\gamma\) values. (**c**) F1-score as a function of \(\alpha\) for different \(\gamma\) values.  
The eMD model, tested in audio-only and multi-modal modes, improves recall compared to Wav2 vec-MLP, but its false alarm rate increases. The multi-modal setup of eMD offers a slight recall boost over the audio-only version, with similar false alarm rates, suggesting limited additional benefit. The eMD\({}_{att}\) model, incorporating attention, further enhances recall in both audio-only and multi-modal modes, reducing missed detections despite a higher false alarm rate. The multi-modal eMD\({}_{att}\) configuration performs best, achieving the lowest missed detection rate (14.15%) and highest recall (85.85%) among all models, with an F1-score of 73.39%, surpassing Wav2 vec-MLP's overall score.  
Compared to the baselines, eMD\({}_{att}\) multi-modal approach stands out, leveraging attention and multi-modal data to outperform Wav2 vec-MLP, with notable improvements in recall (it is similar to the hit rate, an important metric for SCD, because it measures how well the model captures actual speaker change points) and reduced missed detections.