# Unsupervised Adaptation of Deep Speech Activity Detection Models to Unseen Domains
## 2 Domain Adaptation
### Unsupervised Domain Adaptation Techniques
#### 2.3.2 Knowledge Distillation  
The knowledge distillation (KD) [51] framework was originally proposed as a model compression method in which two DNNs are involved. These two models are usually known as teacher and student model in an analogy to the education process. The main idea of this philosophy is that the teacher model produces soft labels which are used to train the student model. Consequently, the student model should imitate the predictions of the teacher model. In order to do so, Kullback-Leibler Divergence (KLD) loss between student and teacher distributions is minimised. KLD loss can formulated according to the following expression:  
\[\text{KLD}=-\sum_{i=1}^{I}p_{t}(y_{i}|x_{i})\log(\frac{p_{s}(y_{i}|x_{i})}{p_{ t}(y_{i}|x_{i})})\,, \tag{1}\]  
where \(i\) is the example index, \(x_{i}\) is the input example, \(p_{t}(y_{i}|x_{i})\) is the output posterior probability of the label \(y_{i}\) from teacher model and \(p_{s}(y_{i}|x_{i})\) is the output posterior probability of the label \(y_{i}\) from student model for the same example. As done in most KD methods [52; 53], the teacher model is usually frozen, relying on a pretrained model, in order to reduce complexity. In this case, where only the parameters of the student network need to be optimised, minimising KLD loss expressed in Equation (1) is equivalent to minimising the expression shown in the following equation:  
\[L_{\text{KLD}}=-\sum_{i=1}^{I}p_{t}(y_{i}|x_{i})\log(p_{s}(y_{i}|x_{i}))+const\,, \tag{2}\]  
where \(const\) is a constant term as defined in [54]. As it has just been explained, knowledge is transferred via the minimisation of a loss function whose target is the distribution of class probabilities predicted by the teacher model. This is the output of a softmax function applied on the teacher model logits. However, in most cases, this distribution provides a high probability for the correct class, with the other class probabilities close to zero. In order to address this issue, the authors of [51] introduced the concept of softmax temperature. The probability \(p_{i}\) of the class \(i\) is calculated from the neural network logits \(z\) according to the following equation:  
\[p_{i}=\frac{e^{(\frac{z}{T})}}{\sum_{i}e^{(\frac{z}{T})}}\,, \tag{3}\]  
where \(T\) is the temperature parameter. For the case of \(T=1\) the standard softmax function is obtained. As \(T\) grows, the probability distribution generated becomes softer, providing more information as to which classes the teacher found more similar to the predicted class.  
The proposed experimental setup for KD based domain adaptation is shown schematically in Figure 1. It can be seen that both models--teacher and student--receive target data examples \(\mathbf{X}_{t}\) as input. Predictions from both models are obtained via softmax activations using the same temperature parameter \(t\). Soft predictions from the teacher network are used to transfer knowledge to the student network, aiming at mirroring those predictions using the mentioned KLD loss. In this process, the teacher model is frozen and only parameters of the student model are updated. In order to test the system, the teacher model is discarded, and final predictions are obtained using the student model with a standard softmax activation with \(T=1\). As it is implemented, this version of KD can be interpreted as a soft version of the pseudo-labelling method previously described.  
Several examples of the use of KD techniques for solving the domain adaptation task can be found in the literature. The authors of [55] apply KD to improve acoustic models for automatic speech recognition (ASR) in application-specific conditions. Something similar is done in [56], that uses KD algorithms to improve ASR performance in noisy environments. Concerning the SAD task, we can also see several examples of the teacher student architecture being used in domain adaptation solutions [57; 58].