# Speech Recognition for Task Domains with Sparse Matched Training Data
## 2 Related Work
### Teacher-Student Learning for Domain Adaptation  
The performance of ASR degrades significantly when there is a mismatch between the training and real test environments. The most intuitive and commonly used solution is to adaptively train a well-trained source domain model to target domain data [8; 32]. However, the disadvantage of domain adaptation is that it requires a considerable amount of labeled target domain data. Thus, it is costly and not suitable for domains where it is difficult to collect large amounts of data.  
To handle these issues, teacher-student (T/S) learning was proposed for the domain adaptation of DNN-HMM-based acoustic models [33]. In teacher-student learning, the Kullback-Leibler (KL) divergence between the posterior distributions of the teacher and student networks, given parallel source and target domain data as input, is minimized by updating the model parameters of the student network. The KL divergence between the source and target distributions is as follows:  
\[D_{KL}\!\left(P_{T}(s|\mathbf{x}_{src})\parallel P_{S}\!\left(s|\mathbf{x}_{ tgt}\right)\right)=\sum_{f}\sum_{i}P_{T}\!\left(s_{i}|\mathbf{x}_{src,f}\right)\! \log\!\left(\frac{P_{T}\!\left(s_{i}|\mathbf{x}_{src,f}\right)}{P_{S}\!\left( s_{i}|\mathbf{x}_{tgt,f}\right)}\right), \tag{2}\]  
Figure 1: Graphical illustration of the factorized hierarchical variational autoencoders (VAEs). Grey nodes denote the observed variables, and white nodes are the hidden variables.  
where \(s_{i}\) indicates a state with index \(i\) and \(f\) is the frame index. \(P_{T}\Big{(}s_{i}|\mathbf{x}_{src,f}\Big{)}\) and \(P_{S}\Big{(}s_{i}|\mathbf{x}_{tgt,f}\Big{)}\) indicate the posterior distribution of the teacher and student networks, respectively, while \(\mathbf{x}_{src,f}\) and \(\mathbf{x}_{tgt,f}\) are the source and target inputs to the teacher and student networks.  
T/S-based training using soft labels of the teacher network output showed improved results compared to conventional cross-entropy training, which directly uses the hard label in the target domain [33; 34; 35]. However, to obtain high performance, T/S learning for domain adaptation requires parallel sequences of source and target domain data, which consist of real data and the simulated pair generated from one domain to the other. These are synchronized frame-by-frame. Examples of parallel source/target data in previous T/S learning studies include real clean speech and simulated noisy speech [33], real adult speech and simulated child speech [33], enhanced clean speech and real noisy speech [34] and real close-talk speech and simulated far field speech [35]. The target domains in the studies are noisy speech, child speech, and far-field speech recognition, respectively. The results of previous studies suggest that the performance of T/S learning tends to be influenced by the quality of the simulated data. For example, the experimental results of T/S learning for the noisy speech domain were shown to outperform conventional training; however, in the child speech domain, only limited performance improvements were obtained.