# Accented Speech Recognition Based on End-to-End Domain Adversarial Training of Neural Networks
###### Abstract  
The performance of automatic speech recognition (ASR) may be degraded when accented speech is recognized because the speech has some linguistic differences from standard speech. Conventional accented speech recognition studies have utilized the accent embedding method, in which the accent embedding features are directly fed into the ASR network. Although the method improves the performance of accented speech recognition, it has some restrictions, such as increasing the computational costs. This study proposes an efficient method of training the ASR model for accented speech in a domain adversarial way based on the Domain Adversarial Neural Network (DANN). The DANN plays a role as a domain adaptation in which the training data and test data have different distributions. Thus, our approach is expected to construct a reliable ASR model for accented speech by reducing the distribution differences between accented speech and standard speech. DANN has three sub-networks: the feature extractor, the domain classifier, and the label predictor. To adjust the DANN for accented speech recognition, we constructed these three sub-networks independently, considering the characteristics of accented speech. In particular, we used an end-to-end framework based on Connectionist Temporal Classification (CTC) to develop the label predictor, a very important module that directly affects ASR results. To verify the efficiency of the proposed approach, we conducted several experiments of accented speech recognition for four English accents including Australian, Canadian, British (England), and Indian accents. The experimental results showed that the proposed DANN-based model outperformed the baseline model for all accents, indicating that the end-to-end domain adversarial training effectively reduced the distribution differences between accented speech and standard speech. accented speech recognition; speech recognition; end-to-end domain adversarial training; domain adversarial neural network; domain adaptation; connectionist temporal classification 2021  
11841221184122118412211841221184122118412211841221184122118412211841221184122118412211841221184122118412211841221184122118412211841221841221841221  
a posteriori (MAP) [3] and maximum likelihood linear regression (MLLR) [4] were mainly used up to the early 2010s. In recent years, neural network-based approaches have been widely used for accented speech recognition. The most representative approach is accent embedding, in which accent embedding features are directly fed into the ASR model. The authors of [5] built a standalone network and a multi-task learning network to perform ASR and accent classification simultaneously. The standalone network classified the type of accent and produced frame-level accent embedding features. The output features of the standalone network were then fed into the multi-task learning network as extra input feature vectors. The authors of [6] advanced the approach by applying an end-to-end method to the ASR model. In the real world, it is difficult to collect a large amount of accented speech data to train deep neural networks, which is one of the challenges of accented speech recognition. The domain adaptation technique can be a solution to this problem [7]. This technique aims to enhance less-trained target domain models that are constructed with a small amount of target domain data by utilizing the information learned from the source domain data that can be collected abundantly. In accented speech recognition, standard speech can be regarded as the source domain, whereas accented speech is considered as the target domain. This study applied the Domain Adversarial Neural Network (DANN) as a domain adaptation technique for accented speech recognition. It has been widely used for computer vision studies [8]. The goal of DANN is to learn domain-invariant features to reduce the distribution differences between the source and target domains with the help of domain adversarial training.