Incorporating Context Information into Deep Neural Network Acoustic Models

The introduction of deep neural networks (DNNs) has advanced the performance of automatic speech recognition (ASR) tremendously. On a wide range of ASR tasks, DNN models show superior performance than the traditional Gaussian mixture models (GMMs). Although making significant advances, DNN models still suffer from data scarcity, speaker mismatch and environment variability. This thesis resolves these challenges by fully exploiting DNNs’ ability of integrating heterogeneous features under the same optimization objective. We propose to improve DNN models under these challenging conditions by incorporating context information into DNN training. On a new language, the amount of training data may become highly limited. This data scarcity causes degradation on the recognition accuracy of DNN models. A solution is to transfer knowledge from other languages to the low-resource condition. This thesis proposes a framework to build cross-language DNNs via languageuniversal feature extractors (LUFEs). Convolutional neural networks (CNNs) and deep maxout networks (DMNs) are employed to improve the quality of LUFEs, which enables the generation of invariant and sparse feature representations. Also, we study a parallelization mechanism to speed up training of LUFEs over multiple GPUs. As with GMMs, the performance of DNNs degrades when the mismatch between acoustic models and testing speakers exists. For GMMs, speaker adaptive training (SAT) trains the acoustic models in a normalized space so that the resulting models generalize better to unseen testing speakers. In this thesis, we present a novel framework to perform feature-space SAT for DNN models. We leverage i-vectors as speaker representations to project DNN inputs into a normalized feature space. The DNN model fine-tuned in this new feature space rules out non-speech variability and becomes more independent of specific speakers. Our SAT-DNN approach is a general framework that can be naturally extended to other deep learning models such as CNNs. Also, the recognition results of SAT-DNNs can be further improved by applying model-space DNN adaptation atop of SAT-DNNs during decoding. Environment variability such as noise and reverberation poses challenges to ASR. In real-world applications, a critical part of the variability comes from the varying distance between the speakers and microphones. In the final part of this thesis, we improve the robustness of DNN models by incorporating this distance information. The distance descriptors are derived from a DNN model which is trained on a meeting corpus. With these descriptors, our distance-aware DNNs capture the speakermicrophone distance dynamically at the frame level. Furthermore, on the task of transcribing videos, visual features from the video stream provide additional indication about the acoustic environment. For instance, images from the videos may indicate the scenes (offices, cars, etc.) where the speech data have been recorded. We examine the utility of visual features as additional environment descriptors, and study DNN architectures that can fuse context information from heterogeneous sources.