# The Domain Mismatch Problem in the Broadcast Speaker Attribution Task
## 5 Results
### Broadcast Domain Mismatch in Speaker Attribution  
The results from Table 2 have shown the importance of diarization as well as the impact of individual adjustments of the system to each specific piece of information (subset, show or audio). However, when no oracle calibration is available, systems are likely to degrade.  
The next experiment analyses the two previously evaluated systems, i.e., the direct assignment as well as the indirect assignment one. However, now, we do not apply an oracle  
\begin{table}
\begin{tabular}{l c c} \hline \hline
**Scenario** & **Development DER (\%)** & **Test DER (\%)** \\ \hline Closed scenario & 6.72 & 8.67 \\ Open scenario & 17.27 & 15.16 \\ \hline \hline \end{tabular}
\end{table}
Table 1: DER (%) results for the Albayzin 2020 corpus, including results for both development and test subsets.  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Data Subset** & \begin{tabular}{c} **Subset-Level** \\ **Direct** \\ \end{tabular} & \begin{tabular}{c} **Show-Level** \\ **Direct** \\ \end{tabular} & \begin{tabular}{c} **Audio-Level** \\ **Indirect** \\ \end{tabular} & \
\begin{tabular}{c} **Lev1** \\ **Direct** \\ \end{tabular} &
\begin{tabular}{c} **L** \\ **Indirect** \\ \end{tabular} \\ \hline Dev. subset & 41.91 & **37.45** & 41.27 & **35.88** & 39.89 & **29.09** \\ Eval. subset & 48.19 & **34.87** & 41.70 & **28.10** & 40.31 & **26.54** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Study of the impact of diarization on speaker attribution with oracle calibration. Experiments carried out with direct (without diarization) and indirect assignment (with diarization) systems. Three degrees of calibration generality are shown. AER (%) results for the Albayzin 2020 development and test subsets. Experiment corresponding to the open condition.  
calibration but estimate one according to the whole development subset. Additionally, we also evaluate the hybrid system from Section 4.6, which is supposedly more robust against domain mismatches. The obtained results for these three systems in both closed-set and open-set conditions are included in Table 3.  
The results in Table 3 reveal that systems depending on a tuned calibration (direct and indirect assignment) strongly suffer from degradation with respect to the oracle result in Table 2 (above a relative higher than a 35% relative degradation) when dealing with domain mismatch. Moreover, we also see great differences in behavior between development and test (degradations over a relative 55%), while particular calibrations (Table 2) did not. Furthermore, this loss of performance is characteristic for the open-set condition; loss of performance does appear in the closed set where no calibration is applied. Regarding the hybrid system, its behavior manages to obtain the best results in the open-set condition, obtaining improvements up to a relative 47%, which also outperforms some scenarios with oracle calibration in Table 2. When applied to the closed-set scenario, its performance is also the best one, yet the improvements are not as noticeable (relative 4%) compared to its direct and indirect assignment counterparts. This is because a closed-set scenario does not require any calibration tuning to fit the audio under analysis.