# Unsupervised Adaptation of Deep Speech Activity Detection Models to Unseen Domains
## 4 Results
### Knowledge Distillation Domain Adaptation  
According to the theoretical explanation provided in Section 2.3.2, we aim to perform domain adaptation using the knowledge distillation framework applied to the SAD task. In the following, we describe the training process. First, teacher and student models are initialised using the unadapted baseline model. Teacher model weights are frozen during the entire training, and only student model weights are updated. The output of both models is compared using KLD loss after going through softmax activation with a temperature parameter \(T\), shared for teacher and student models. At inference, the softmax layer is used in its standard form with temperature \(T=1\). Obtained results in terms of AUC and EER using knowledge distillation are shown in Table 5 for various values of the temperature parameter \(T\).  
A decreasing tendency for EER can be observed when increasing the temperature parameter up to \(T=50\), achieving a best EER value of 6.05%. However, this tendency is not consistent for the AUC metric, showing values in between 97.80 and 97.86. When compared to the baseline system, best temperature configuration reports a 8.10% relative improvement on the EER value, yet this improvement in EER only leads to a 0.29% relative improvement on the AUC metric. In general terms, an improvement in performance can  
Figure 5: DET curve and EER (%) on the FS02 development partition using the pseudo-label domain transfer setup training a new neural network from scratch (**left**) and fine-tuning the baseline neural network (**right**), both compared to the baseline system.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Softmax Temperature** & **AUC (\%)** & **EER (\%)** \\ \hline \(T=1\) & 97.79 & 6.41 \\ \(T=10\) & 97.72 & 6.32 \\ \(T=20\) & 97.80 & 6.27 \\ \(T=30\) & 97.80 & 6.25 \\ \(T=40\) & 97.72 & 6.23 \\ \(T=50\) & **97.86** & **6.02** \\ \(T=60\) & 97.70 & 6.33 \\ \hline \hline \end{tabular}
\end{table}
Table 5: AUC (%) and EER (%) on the FS02 development partition using knowledge distillation domain adaptation setup with various softmax temperature values compared to the baseline system.  
be observed using the temperature softmax activation as argued by [51], however this improvement is limited.  
Results in Table 5 are complemented with those presented in Figure 6. In this Figure, we present the DET curve and EER for some of the best performing knowledge distillation systems compared to the unadapted baseline system.  
As observed in Figure 6, unlike the pseudo-labelling strategy, knowledge distillation seems to be able to decrease the EER point on the DET curve by using a large temperature parameter. However, it can also be seen that all curves are very close to each other; this is translated in AUC values very similar to those obtained by the baseline system. In general terms, it can also be observed that knowledge distillation does not correct the baseline system tendency to provide high FNR values. This may be due to the fact that KLD loss makes the student network mimic the predictions of the teacher network, so probability distributions and the relations between speech and non-speech observations should remain similar.