# Improvement in Automatic Speech Recognition of South Asian Accent Using Transfer Learning of DeepSpeech2

Authors: Muhammad Ahmed Hassan; Asim Rehmat; Muhammad Usman Ghani Khan; Muhammad Haroon Yousaf
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

Automatic speech recognition (ASR) has ensured a convenient and fast mode of communication between humans and computers. It has become more accurate over the passage of time. However, in majority of ASR systems, the models have been trained using native English accents. While they serve best for native English speakers, their accuracy drops drastically for non-native English accents. Our proposed model covers this limitation for non-native English accents. We fine-tuned the DeepSpeech2 model, pretrained on the native English accent dataset by LibriSpeech. We retrain the model on a subset of the common voice dataset having only South Asian accents using the proposed novel loss function. We experimented with three different layer configurations of model to learn the best features for South Asian accents. Three evaluation parameters, word error rate (WER), match error rate (MER), and word information loss (WIL) were used. The results show that DeepSpeech2 can perform significantly well for South Asian accents if the weights of initial convolutional layers are retained while updating weights of deeper layers in the model (i.e., RNN and fully connected layers). Our model gave WER of 18.08%, which is the minimum error achieved for non-native English accents in comparison with the original model.


---

## Body

## 1. Introduction

Automatic speech recognition (ASR) is a key component in making human-computer interaction (HCI) hassle-free because it is the most interactive and convenient mode of communication between automated systems and humans [1]. The interaction between human and voice-based systems is mostly accomplished in English language. Some of the common applications of voice-controlled systems in AI world are chatbots, humanoid robots, healthcare systems [2], self-driving cars, surveillance systems, industrial robots, and many more. From these applications, chatbots are the most widely used till date. They are everywhere around us. Regardless of region state or country, everyone is using it. Google Assistant [3], SIRI [4], Cortana [5], and Alexa [6], all of them are helping humans in each step of their daily life [7]. Form booking an appointment for barbers’ shop to remainder for daily grocery items. Over half of the world’s population now has a mobile phone of which 17% of them are smartphones [8], which means around a billion smartphones and 700 million windows users [9] out there in this world. And all of them are equipped with chatbots. Mobile phones come with SIRI or Google Assistant and Windows users have Cortana.Similar to chatbots, self-driving cars have similar issues. According to Google Waymo, autonomous vehicles have reached 8 million test miles in July 2018 [10]. This clearly states, how close humans are to bringing 5th level of autonomous vehicles to reality. These autonomous systems are designed to operate through voice commands with English according to the British Council, 1.75 billion people in this world speak English [11]. America has the biggest share with around 268 million native English-speaking people [12], UK is second with 59.6 million speakers [13], Europe is third with collectively 70 million people, who speak English as a native language, [14] and Canada is fourth with 19 million native speakers [15]. Although the abovementioned countries have more population of native English-speaking people, but there are several non-native English speakers, present in these countries as well. Apart from this, there is number of counties, which uses English as a second language and their accent is different than that of native English speakers.For instance, Pakistan and India are holding the biggest share of non-native English speakers, i.e., around 88.6 million people in Pakistan and 125 million in India speak English as a secondary language [16]. Due to regional differences, South Asians and Gulf countries vocal accent for English is different from native English-speaking countries [17]. This highlights the issue of usability, for the voice-controlled systems with dissimilar accents. Which eventually generates hurdles in the use of discussed human voice-controlled systems. Although ASR is being used widely, it is not flexible enough for non-native English accents—thus, 290 million people are unable to use its applications properly.The motivation behind this research area is given as follows:(i)
Make more accurate the English ASR for non-native English speakers(ii)
Use existing ASR system’s accuracy and make it more robust by adding an additional pipeline for non-native English Speakers.(iii)
Make model learnable for a small amount of dataFor the last two decades, Hidden Markov’s model (HMMs) and Gaussian mixture models (GMMs) were very effective in improving the recognition accuracies of ASR. However, in recent few years, deep neural network (DNNs) [18] has replaced GMMs although the remaining part of GMM-based recognition architecture is still kept for several experiments. These systems are called hybrid ASR systems [19] because they use classic HMM/GMM-based architecture and after the training, they replace the GMM with DNN. Likewise, recurrent neural networks (RNNs) are used as well in a similar manner for language modeling.There is a plethora of work regarding ASR. For most of the ASR systems, models are trained using a native English accent. Some of the described models have achieved above 90.0% accuracy. Like Google’s model for ASR [20] can be as accurate as the human itself in some cases. They claimed of achieving 95.0% of accuracy. But this model is not performing well for non-native English accents until now. Research and development in ASR continuously getting better [21] with large-scale pieces of training, deeper network architectures, and reduced word error rate (W.E.R) have been providing efficient results for ASR. Microsoft AI and Research Lab published research shows 5.1% W.E.R on 2000 switch-board evaluation set by adding additional acoustic model architecture to their system [22]. They called it CNN-bidirectional long short-term memory networks (CNN-BLSTM). Their work clearly reflects closeness to human efficiency. But the problem still occurs when it comes to South Asian accents.In deep learning, the learning of the model is directly proportional to the amount of data. The more data model has for training, it learns more and makes general decision boundaries. Sometimes, if the model needs some modification in class labels then it requires training from scratch. The solution to this problem was provided by transfer learning [23]. Through transfer learning, the learning of the model can be transferred to new similar problems with some modifications to the last layers of the model. We can use the trained weights of the model and find the best weights for the last layer, that is, called the classification layer.Recently, DeepSpeech2 [24] provides a deep learning-based architecture that gave promising results in English and Mandarin languages. The deep speech architecture consists on 1D convolutional layers, RNN layers, and fully connected layers. The DeepSpeech2 architecture gives awesome results for two very different languages. It means it has the capability to learn the features of different languages. So, we decided to evaluate DeepSpeech2 on non-native English speakers and improve the quality of DeepSpeech2 by transfer learning.The most basic limitation in training these models is the limited available dataset, i.e., non-native English speakers’ dataset is limited and not widely available. Consequently, most of the models are unable to recognize non-native English accents. To cover this gap, we proposed a system for the recognition of English language, specifically for non-native English accent speakers. Our system will recognize and generate a transcription of human voice using deep learning model named DeepSpeech2 [25]. Our proposed solution will address the following points to reduce the word error rate (W.E.R) on South Asian accents for English language automatic speech recognition (ASR).(1)
We propose a hybrid model based on DeepSpeech2 with two pipelines that learn both English and non-native English accent.(2)
We propose a novel loss function that optimizes the model’s weights better.(3)
We achieve 18.08% for non-native English and 7.0% WER for English accent. 2 [26] model by fine-tuning its model on using non-native English accent dataset.


## 2. Literature Review

Automatic speech recognition is not new to this era of 4th industrial revolution wave. It all started in the middle of 19th century. In 1950, researchers from bell labs build a system named “Audrey” [27] to recognize a digit for single person [28]. Audrey was a six-foot-high relay rack, capturing considerable power in addition to streams of cable. It was capable of recognizing digits from speech using phonemes.Although in the 1950s computer systems were not so good, they had limited computational speed and memory. But Audrey was perfect in recognizing digits from 1 to 9 with more than 90% of accuracy [29]. It also produces above 70% accuracy in the case of some selected unknown speakers. But Audrey was not comfortable with unknown voices, which means lesser accuracy. From 1960 to 1970, most of the exploration and phonetic segmentation work was completed. Some major techniques used for ASR in 60 s were brute force approach and template. It was good in results but it is hard to scale. A big breakthrough in that era was speech understanding research (SUR), the project of DARPA [30].Later on, in the 1980s some of remarkable discoveries were found. First appearance of hidden Markov models (HMM) [31] changed the way of speech recognition. With HMM, neural networks also played their role. Layered feed-forward networks with sigmoid function are used to train the model for speech recognition [32]. A three-layer net was constructed and a back-propagation learning procedure is used to train the network. Results of these neural nets are better than HMMs. Time delay neural network (TDNN) achieved 1.5% of word error rate over HMM’s 6.3% of word error rate [33]. With the development of HMM and neural networks, many breakthroughs were achieved. DARPA started new speech projects, HMM become popular. Rabiner at bell labs performs well using HMM, AT&T performed the first large-scale deployment of speech recognition named (voice recognition, call processing) VRCP [34, 35]. Automated systems were deployed, in the late 1990s United Airlines launches an automatic flight information system.In the last decade of research and development in the area of ASR, sensor networks [36] computer vision [37], and natural language processing grows rapidly. Deep learning innovation played a vital role in it. In a recently published Microsoft research [38], very impressive results had been recorded. LSTM was preferred on RNN-LMs to achieve proficiency in reducing W.E.R. Models were built using CNTK. Human versus machine errors is analyzed, which indicates substantial equivalence. In this research, NIST 2000 dataset was used, which produces 4.9% of word error rate [39]. But NIST 2000 dataset was originally recorded from calls with native English accent. That is why it is not very much accurate with South Asian accent.Kadyrov et al. [40] proposed an ASR based on spectrogram images of speech signals. They achieved 98.34% accuracy. But they used a self-generated dataset. They did not evaluate the model on different accents of English speakers, we consider the different accents of the English speakers and evaluate them on standard benchmark.Another method to gain efficiency in automatic speech recognition was through active learning. In it, a gradient-based active learning method was used [41]. Active learning aims to label only the most informative data. It helps in reducing labeling costs. In a result, it outperformed the confidence score method used in ASR. Deep learning approaches have achieved significant accuracies in ASR. CNN is key player in achieving these accuracies. Mostly less than 10 layers CNN architecture is used to design models for learning features. But Yisen Wang proposed deep and wide CNN architecture. This architecture is known as RCNN-CTC, it consists of residual connections and connectionist temporal classification loss function [25]. Resulting in 14.92% W.E.R on WSJ dev93 and 6.52% W.E.R on Tencent chat datasets.One of the core difficulties in automatic speech recognition is noise. Because real-time speech data are filled with different noises like background noises, sampling rates, and codec distortion. Google recently published research to overcome this issue [42]. They trained their model on 162,000 hours of speech. Their goal is to make a generally robust system. Previously most of the models are domain robust, like noise. Google applied various techniques to ensure the robustness of the system by using multiple codecs for encoding inputs in the presence of background noise [43]. More interestingly their model performs very well in new unseen conditions. Their multidomain model trained on 10 hours of data outperformed a model trained for 700 hours of speech data on a new domain only [44].The survey of previous ASR methodologies is also described in Table1.Table 1
Literature survey of previous methodologies.
YearPaperMethodologyLanguageAccentWER2021[45]Unsupervised CNNEnglish—12.832021[46]RNN encoder-decoder + CTCEnglish—8.32021[47]Quantifying biasDutch—16.852022[48]TransformersEnglish—39.92022[49]Semisupervised learningEnglish—9.42022ProposedMulti pipeline feature extractor learningEnglishNative + South Asian7All of the abovementioned outstanding results provide an overview of ASR history and development. But there is one common problem with all these systems, which is non-native English accents. All datasets, used for those training, are recorded or collected from native English-speaking sources, which are different from South Asian English accents. Our proposed system is for the recognition of English language specifically for the non-native English accent as Asian accent is different from native English language, by reducing the (W.E.R) on South Asian accent. Our framework will perceive and create interpretation of human voice utilizing DeepSpeech2, where a few adjustments are suggested in the system layers.


## 3. Methodology

This paper makes a contribution toward automatic speech recognition for English language in native English and non-native English accents. This research work is inspired by DeepSpeech2 model for English and Mandarin languages. The whole system architecture is shown in Figure1. The first step involves the preprocessing of the dataset, the second part is feature extraction from audio signals, and the third part is proposed two pipelined CNN-RNN models. The last stage of the system is the decoder, which is used for postprocessing of the predicted transcriptions. Each module of the proposed system is explained below:Figure 1
System architecture of the proposed system.


### 3.1. Data Preprocessing

Common voice (CV) dataset was not recorded in a controlled environment, which means that volunteers used their own devices for the recording of CV dataset. The recording was completed with the help of microphones and Internet browsers. Due to the fact that the recording took place in an uncontrolled environment, too much noise was introduced in the background of recorded audio, for instance, the distortion at the beginning of the audio, similar to the noise generated by the microphone, when plugged into the port. Moreover, it has empty gaps (unnecessary silences) between the words and sentences, for example, the speaker starts speaking after 0.5 to 1.5 second delay, and sometimes takes a long stay in between two sentences while reading paragraphs. Thus, CV dataset was useless in raw form and required tons of cleaning and preprocessing. We have cleaned all of South Asian separated audio files by removing the noise and deleting silences between sentences. We employed a self-generated Algorithm 1 to scan each audio file and perform the following activities on it to make sure data are useable for training and prediction. The preprocessing steps include as follows:(1)
Deletion of empty audio files(2)
Elimination of unnecessary silences between the sentences and words(3)
Removal of loud noisy sounds from the beginning of recordings(4)
Extraction of audio file in the FLAC format as per the requirement of networkAlgorithm 1: Dataset preprocessing algorithm.
Read CSV of common voice datasetChoose “Pakistani,” “Indian,” “Dutch,” and “Sri-Lankan” accent rows from country columnSelect “filename” columns from CSV fileCreate a dictionary for JSON file writingfori in filename doflacConvert(i)SilenceRemover(i)Duration(i)Finding transcript with respect to audio filefor lineIndex in range(x) dotranscript = split[lineIndex]Write JSON filedata = {“audio_filepath”: “1.FLAC,” “duration”: duration, “text”: transcript}with open(‘json-other-train.txt,” “a”) as outfile:json.dump(data, outfile)end forend forflacConvert(i) (Convert mp3 -> FLAC)Calculate audio file lengthCalculate audio file sample ratesCalculate duration = audio file length/sample rateCalculate loudnessCalculate peak amplitudeSplitting audio file into chunks with silence > 0.5 second (considering it silent if quieter than −16 dBFS)for chunk in enumerate (chunks) doList.Append(chunk)end forIn order to remove silence and loud noisy sounds, we used zero crossing rate (ZCR) methodology [50]. It is observed that speech section of audio file computes a low zero crossing value and in silent parts, it gives a higher zero crossing value [51]. It is because of the fact that zero crossing count indicates frequency, which is concentrated by energy in the spectrum of voice signals. Vocal sounds are produced by repeated flow of air through the glottis by excitation of the vocal tract, which usually generates a low zero crossing count. Whereas speech other than voice is formed by a narrow vocal tract to cause turbulent airflow that will eventually result in noisy sound and outputs a high zero crossing count.(1)Z.C.R=1T−1∑t=1−1IR<0St,St−1,where S = Signal, T = Length of Signal, T = time,After filtering audio files, we saved it to FLAC format because FLAC files are better in audio quality than mp3. The visual representation of audio signal before and after preprocessing is shown in Figure2.Figure 2
Audio signal processing to remove unnecessary silence and noise.


### 3.2. Feature Extraction

The MFCC features are used as input for the model. In the last few decades, these features show very excellent results in automatic speech recognition, semantic analysis through speech, gender classification, and emotion recognition through speech. MFCC features are calculated by the given equation as follows:(2)Cn=∑j=1JlogEjcosnj−12πm,forn=1,2,3,…,k,here E (J) is DFT bin energies that are obtained by the following equation:(3)EJ=∑Wkfxf2.The proposed model also implements the attention mechanism to weight the extracted features. The functionality of the attention layer can be expressed through the following equation:(4)Ci=∑i=1naifi,here ai represents the attention score of ith features and fi is the actual value of ith feature.


### 3.3. Proposed Network

The proposed network architecture of DeepSpeech2 has been used with an extra pipeline for non-native English accents and a novel loss function described in the loss function section. The DeepSpeech2 architecture contains two 1D convolution layers, three bidirectional RNN layers, and one fully connected layer. The 1D convolution layers extract the features from the signal by convolving the 1D-kernel over the Mel frequency cepstral coefficients (MFCC) features, where RNN layer is a state full layer, that extracts the temporal information from the features extracted by convolution layers. The fully connected layer then predicts the text using the features extracted by both convolutional and RNN layers. The proposed model consists of two pipelines of convolution layers. These two pipelines are introduced to extract features of English and non-native English accents. As the DeepSpeech2 model was trained on the English accent dataset so, we used the DeepSpeech2 model with its trained weights to extract English accent features. The structure of the second pipeline is same as DeepSpeech2 model having initial DeepSpeech2 weights that are further fine-tuned using non-native English accent dataset.MFCC features have been used as input for both pipelines of model. First convolution layer contains 32 filters of size 11 × 41 × 1 with a stride size of 3 × 2. Second convolution layer contains 32 filters of size 11 × 21 with a stride size of 1 × 2. Both convolution layers perform padding to avoid the down sampling of data. Three bidirectional RNN layers are stacked followed by convolution layers. The last bidirectional RNN layer of both pipelines out 2,048 features. The features from both pipelines are concatenated and further passed to FC1 layer having 4,096 neurons. Now the proposed network extracts double features (2048 + 2048 = 4096) than deep speech features (2048). The excess of features leads the network to overfit. To avoid overfitting dropout layer is used after FC1 layer. An attention layer is also introduced after the dropout layer for weighting the features from English and non-native English accent pipelines. The weighted features are further passed to FC2 layer having a number of neurons equal to vocabulary size. The softmax layer is used to predict the probability of each character. The probabilities of each character can be calculated by the following equation (5):(5)pc=ec∑k=1kec,here p (c) shows the probability of z class and ec shows the score of z class that is produced by FC2 layer. K represent the size of the vocabulary.The architecture of the proposed network is shown in Figure3.Figure 3
Proposed network architecture.


### 3.4. Decoder

The transcriptions produced by the model are mostly correct without English language constraints like spacing and sentence boundary, etc. To handle this problem a language model is used by Amodei [24]. We extend the vocabulary of LibriSpeech dataset [52] with the common voice dataset’s transcriptions. A decoder is developed using a language model and vocabulary that accepts the predicted transcriptions from RNN model and produced the transcription that satisfies the English language constraints as shown in Table 2.Table 2
Sample transcriptions generated by proposed model and decoded by the language model.
RNN outputDecoder outputKids are playing foot balling roundKids are playing football on the groundShe is trying together toysShe is trying to get her toysBirds flying roupsBirds fly in groups


## 3.1. Data Preprocessing

Common voice (CV) dataset was not recorded in a controlled environment, which means that volunteers used their own devices for the recording of CV dataset. The recording was completed with the help of microphones and Internet browsers. Due to the fact that the recording took place in an uncontrolled environment, too much noise was introduced in the background of recorded audio, for instance, the distortion at the beginning of the audio, similar to the noise generated by the microphone, when plugged into the port. Moreover, it has empty gaps (unnecessary silences) between the words and sentences, for example, the speaker starts speaking after 0.5 to 1.5 second delay, and sometimes takes a long stay in between two sentences while reading paragraphs. Thus, CV dataset was useless in raw form and required tons of cleaning and preprocessing. We have cleaned all of South Asian separated audio files by removing the noise and deleting silences between sentences. We employed a self-generated Algorithm 1 to scan each audio file and perform the following activities on it to make sure data are useable for training and prediction. The preprocessing steps include as follows:(1)
Deletion of empty audio files(2)
Elimination of unnecessary silences between the sentences and words(3)
Removal of loud noisy sounds from the beginning of recordings(4)
Extraction of audio file in the FLAC format as per the requirement of networkAlgorithm 1: Dataset preprocessing algorithm.
Read CSV of common voice datasetChoose “Pakistani,” “Indian,” “Dutch,” and “Sri-Lankan” accent rows from country columnSelect “filename” columns from CSV fileCreate a dictionary for JSON file writingfori in filename doflacConvert(i)SilenceRemover(i)Duration(i)Finding transcript with respect to audio filefor lineIndex in range(x) dotranscript = split[lineIndex]Write JSON filedata = {“audio_filepath”: “1.FLAC,” “duration”: duration, “text”: transcript}with open(‘json-other-train.txt,” “a”) as outfile:json.dump(data, outfile)end forend forflacConvert(i) (Convert mp3 -> FLAC)Calculate audio file lengthCalculate audio file sample ratesCalculate duration = audio file length/sample rateCalculate loudnessCalculate peak amplitudeSplitting audio file into chunks with silence > 0.5 second (considering it silent if quieter than −16 dBFS)for chunk in enumerate (chunks) doList.Append(chunk)end forIn order to remove silence and loud noisy sounds, we used zero crossing rate (ZCR) methodology [50]. It is observed that speech section of audio file computes a low zero crossing value and in silent parts, it gives a higher zero crossing value [51]. It is because of the fact that zero crossing count indicates frequency, which is concentrated by energy in the spectrum of voice signals. Vocal sounds are produced by repeated flow of air through the glottis by excitation of the vocal tract, which usually generates a low zero crossing count. Whereas speech other than voice is formed by a narrow vocal tract to cause turbulent airflow that will eventually result in noisy sound and outputs a high zero crossing count.(1)Z.C.R=1T−1∑t=1−1IR<0St,St−1,where S = Signal, T = Length of Signal, T = time,After filtering audio files, we saved it to FLAC format because FLAC files are better in audio quality than mp3. The visual representation of audio signal before and after preprocessing is shown in Figure2.Figure 2
Audio signal processing to remove unnecessary silence and noise.


## 3.2. Feature Extraction

The MFCC features are used as input for the model. In the last few decades, these features show very excellent results in automatic speech recognition, semantic analysis through speech, gender classification, and emotion recognition through speech. MFCC features are calculated by the given equation as follows:(2)Cn=∑j=1JlogEjcosnj−12πm,forn=1,2,3,…,k,here E (J) is DFT bin energies that are obtained by the following equation:(3)EJ=∑Wkfxf2.The proposed model also implements the attention mechanism to weight the extracted features. The functionality of the attention layer can be expressed through the following equation:(4)Ci=∑i=1naifi,here ai represents the attention score of ith features and fi is the actual value of ith feature.


## 3.3. Proposed Network

The proposed network architecture of DeepSpeech2 has been used with an extra pipeline for non-native English accents and a novel loss function described in the loss function section. The DeepSpeech2 architecture contains two 1D convolution layers, three bidirectional RNN layers, and one fully connected layer. The 1D convolution layers extract the features from the signal by convolving the 1D-kernel over the Mel frequency cepstral coefficients (MFCC) features, where RNN layer is a state full layer, that extracts the temporal information from the features extracted by convolution layers. The fully connected layer then predicts the text using the features extracted by both convolutional and RNN layers. The proposed model consists of two pipelines of convolution layers. These two pipelines are introduced to extract features of English and non-native English accents. As the DeepSpeech2 model was trained on the English accent dataset so, we used the DeepSpeech2 model with its trained weights to extract English accent features. The structure of the second pipeline is same as DeepSpeech2 model having initial DeepSpeech2 weights that are further fine-tuned using non-native English accent dataset.MFCC features have been used as input for both pipelines of model. First convolution layer contains 32 filters of size 11 × 41 × 1 with a stride size of 3 × 2. Second convolution layer contains 32 filters of size 11 × 21 with a stride size of 1 × 2. Both convolution layers perform padding to avoid the down sampling of data. Three bidirectional RNN layers are stacked followed by convolution layers. The last bidirectional RNN layer of both pipelines out 2,048 features. The features from both pipelines are concatenated and further passed to FC1 layer having 4,096 neurons. Now the proposed network extracts double features (2048 + 2048 = 4096) than deep speech features (2048). The excess of features leads the network to overfit. To avoid overfitting dropout layer is used after FC1 layer. An attention layer is also introduced after the dropout layer for weighting the features from English and non-native English accent pipelines. The weighted features are further passed to FC2 layer having a number of neurons equal to vocabulary size. The softmax layer is used to predict the probability of each character. The probabilities of each character can be calculated by the following equation (5):(5)pc=ec∑k=1kec,here p (c) shows the probability of z class and ec shows the score of z class that is produced by FC2 layer. K represent the size of the vocabulary.The architecture of the proposed network is shown in Figure3.Figure 3
Proposed network architecture.


## 3.4. Decoder

The transcriptions produced by the model are mostly correct without English language constraints like spacing and sentence boundary, etc. To handle this problem a language model is used by Amodei [24]. We extend the vocabulary of LibriSpeech dataset [52] with the common voice dataset’s transcriptions. A decoder is developed using a language model and vocabulary that accepts the predicted transcriptions from RNN model and produced the transcription that satisfies the English language constraints as shown in Table 2.Table 2
Sample transcriptions generated by proposed model and decoded by the language model.
RNN outputDecoder outputKids are playing foot balling roundKids are playing football on the groundShe is trying together toysShe is trying to get her toysBirds flying roupsBirds fly in groups


## 4. Experiments and Results

In this section, the experimental setup of model training, evaluation measure, and results are discussed.


### 4.1. Experimental Setup

The training is done for three different layer configurations of the network. In configuration A, the convolutional layer freezes, and learning of RNN layer and FC layers takes place by modifying their weights. Configuration B is made by freezing the convolution layer and FC layer and modifying weights of RNN layer. In configuration C, the RNN and FC layers are frozen and let the convolution layers learn. These three configurations are shown in Table3.Table 3
Different configuration settings for non-native English accent pipeline.
ConfigurationFreeze layerTraining layerANoneConv, RNN, and FCBConv and FCRNNCConvRNN and FCWhile training, one pipeline for non-native English accent was trained and the other pipeline was used as is with pretrained weights of the original DeepSpeech2 model. The English accent pipeline was freezed by setting the learning rate 0 for all layers. The proceeding implementation details for non-native English accent pipeline.The training of model is done using ReLU activation in all of the layers. The proposed loss function as discussed in Section 4.1.1 is used as a criterion. Stochastic gradient descent (SGD) optimizer is used with dynamic learning rate, starting froml = 1 × e−3 with decay rate of 1 × e−1. The model is trained for 200 epochs. The experiments are performed on the system having Nvidia’s 1080 Ti GPU having 3584 Cuda cores and 11 GB cud memory. The system contains 16 GB DDR3 RAM and a 2.8 quad-core processor. The total dataset is split in a 70–30 ratio. A total of 70% of the data are utilized for learning the model and the remaining 30% of data are used to evaluate the learning of the model. The model’s architecture is designed and trained on paddle paddle framework. Some of the main python libraries for data manipulation and audio signal processing are used such as Pandas, Numpy, SciKit, and PyAudio.The model hyperparameters are shown in Table4.Table 4
Parameters and their values while implementation.
ParametersValueOperating systemUbuntu 18.04Frame workPaddle paddleLanguagePython 3.7CPUCore-i7 (7th gen.)RAM16 GB (DDR3)GPU1080 Ti (11 GB memory, 3584 cores)Batch size256Epochs200Drop out0.4Learning ratel = 1 × e−3Loss functionProposed Section 4.1.1OptimizerSGD


#### 4.1.1. Loss Function

Most existing ASR models were optimized using some cost functions that take the predicted outputy′ and ground truth y. The difference y and y′ is used to optimize the model parameters. The proposed optimization technique intends to reduce the feature gap between the South Asian and European accents. So, the proposed loss calculation function uses the features of South Asian and European accent audio from FC2 layer, and the mean square difference of these features is used as a loss to optimize the model parameters as shown in the following equation:(6)da,e=1Na1−e12+a2−e22+⋯+ai−ei2+⋯+an−en2,here a is feature vector of South Asian accent and e is feature vector of European accent on the same transcription and N is number of samples. The objective of this loss function is to decrease the difference between the feature vector of South Asian and European accents to reduce the accuracy gap in ASR for South Asian and European accents.


### 4.2. Dataset

We have used common voice [53] dataset for fine-tuning of DeepSpeech2 [54] models. common voice dataset was recorded for more than 18 languages by Mozilla for the purpose of research. It consists of total 1087 hours of audio files, from which 780 hours were validated with transcription. This dataset was recorded with both male and female voices with the ratio of 47% and 11% at a sampling frequency of 16 kHz. A detailed description of dataset accent according to the region is listed in Table 5.Table 5
Percentage of different English accents in common voice dataset.
Share (%)Accent23United States English9England English4India and south Asia (India, Pakistan, and Sri Lanka)3Canadian English3Australian English1New Zealand English1Southern African (South Africa, Zimbabwe, and Namibia)1Scottish EnglishAs stated earlier, we are focusing on non-native English accents, so we used a subset of this dataset by filtering out Pakistani, Indian, Dutch, and Sri-Lankan speaker’s recordings, with the help of Algorithm 1. Because accent variation is affected by the geographical area in which the speaker grows up and lives as well as by factors such as social class, culture, education, and working environment. All of these factors have an impact on the accuracy of the automatic speech recognition system. After splitting desired recorded files, we got a total of 10,219 audio files with an average playtime of 5 seconds. Table6 shows the further splitting of training and testing classes accordingly with respect to gender.Table 6
Total number of audio files in training and test set of non-native English accents with respect to gender.
UtterancesTraining setTest setMale4,4441,112Female3,730933Total8,1742,045


### 4.3. Evaluation Measures

The model evaluation parameters that we have selected are word error rate (WER), match error rate (MER), and word information rate (WIR).


#### 4.3.1. Word Error Rate

WER is one of the most common evaluation parameters for ASR models and it provides a good comparison between the results of our proposed model and other related work done so far. The WER tells the rate of error in the transcript generated by ASR by comparing it to the original transcript. It can be calculated by the following equation:(7)W.E.R=S+I+DN,where N is the total number of words spoken in the original transcript.(8)N=S+D+H,here, S is the number of substitutions. I is the number of insertions, D is the number of deletions, and H is the total number of hits, i.e., correctly transcribed words.


#### 4.3.2. Match Error Rate

Match error rate tells the probability of given input-output word matches being incorrect. It can be calculated by equation.(9)M.E.R=S+I+DH+S+I+D=1−HN.Unlike in WER, hereN is sum of all four terms.(10)N=H+S+I+D.


#### 4.3.3. Word Information Loss

Word information loss gives the probability of any input word is matched with an equal output word and vice versa. It can be calculated by equation.(11)W.I.L=H2H+S+IH+S+D.All three evaluation metrics represent the errors and loss in the output, hence, the lesser the value is, the better the model predicts. However, WER is not an actual percentage as it has no upper bound limit because of the insertion I parameter. So, WER can only be used to compare different models while MER and WIL can be interpreted as how well the model performs.


### 4.4. Results

Before any modifications happened in DeepSpeech2, we loaded pretrained model on LibriSpeech dataset. As the majority of speakers of this dataset were from US, the model achieved WER of around 6% on the test set on US English accents. As the model was trained on American people and the test was also containing American people, that is why we obtained 6% WER. However, the same model, when evaluated using common voice dataset, gave a drastically high WER of 43% for South Asian accents. The reason for 43% WER is pretrained model of DeepSpeech2, which is not trained for non-native American or non-native English guys, that is why we use parallel pipeline for processing for the non-native English peoples. This is called a learning network. When it finds the input of non-native speakers for English it learns and updates its weights accordingly, meanwhile, we have frozen the learning rate for the freeze network, hence, we can save the performance of our model for the native English users and weights would not be changed for this pipeline. Whereas if the user is not-native English speaker then the learning network will entertain that user and update the weights. That is why our model is working better than other models due to its learning controls.The training of DeepSpeech2 is done in three different layer configurations as mentioned in Table3. The loss comparison graph of configurations A, B, and C is shown in Figure 4. The purpose of experimenting through these configurations is that we want to make DeepSpeech2 perform better for South Asian accents by transfer learning. We experimented with different dropout ratios and the best results were achieved with a dropout ratio 0.7. All the results shown here of different configurations have the same dropout ratio of 0.7.(1)
For configuration A, where all layers were learned using a common voice dataset, the model achieved W.E.R of 35.35% on the validation set. The training and validation cost of this modification is shown in Figure5(a).(2)
For configuration B, the weights of RNN layer have been learned by freezing the both CNN and FC layers. In this configuration, the model retained its low-level features learned from LibriSpeech dataset and learned only the new high-level features from the common voice dataset. The WER achieved on the validation set is 20.419%. Training and validation cost for these configurations is shown in Figure5(b).(3)
Finally, for configuration C, we learned the weights of RNN and FC layers by freezing the convolutional layer. W.E.R achieved on the validation set is 18.0859%. The cost of training and validation is shown in Figure5(c).(4)
We performed fine tuning on both the European accent and South Asian accent datasets. The results deduced after fine-tuning showed that it behaved differently for both datasets, as for European accent the W.E.R increased from 6.8% to 7.0% whereas for South Asian accent W.E.R is reduced from 51% to 18.08%.Figure 4
The image shows a calculated loss after training and validation at config A, config B, and config C.Figure 5
Training is done for 200 epochs. The cost or loss of training in different configurations is shown. (a) The trend of loss function for training and validation of the model in configuration A, (b) the trend of loss function for training and validation of model in configuration B, and (c) the minimum loss for training and validation of model in configuration C.
(a)(b)(c)The WER, MER, and WIL rate of the model on test set are shown in Table7.Table 7
Results of all three-layer configurations of the proposed model in terms of WER, MER, and WIL.
ConfigSouth Asian accentWestern accentWER (%)MER (%)WIL (%)WER (%)MER (%)WIL (%)A35.3529.6830.9810.3612.6511.22B20.41918.5419.359.688.699.85C18.085915.3615.257.06.87.2We contrasted our DeepSpeech2 algorithm to Apple Dictation, Bing Speech, Google Speech API, and wit.ai, which are all for profit speech technologies. Our test is intended to monitor success in noisy situations. This circumstance complicates the evaluation of web audio APIs: whenever the SNR is just too small or, in certain situations, whenever the phrase is too lengthy, such algorithms will provide no results. As an outcome, we limit our analysis to phrases under that all algorithms gave a not-a-void outcome. Table8 shows the outcome of assessing each system on our test files.Table 8
The comparison between nonoptimized and optimized models with respect to WER and CER.
ModelNonoptimizedOptimizedWERCERWERCERDeepSpeech215.574.52——DeepSpeech2 KENLMc10.463.6810.452.96DeepSpeech2 KENLMo10.753.7910.662.89DeepSpeech2 and KENLM (c + O)9.93.619.912.80The comparison of the proposed model and DeepSpeech2 model is shown in Table9.Table 9
Comparison of DeepSpeech2 and proposed model’s WER on English and non-native English accent.
ModelWER (English accent) (%)WER (non-native English accent) (%)DeepSpeech 2643Proposed718.08


## 4.1. Experimental Setup

The training is done for three different layer configurations of the network. In configuration A, the convolutional layer freezes, and learning of RNN layer and FC layers takes place by modifying their weights. Configuration B is made by freezing the convolution layer and FC layer and modifying weights of RNN layer. In configuration C, the RNN and FC layers are frozen and let the convolution layers learn. These three configurations are shown in Table3.Table 3
Different configuration settings for non-native English accent pipeline.
ConfigurationFreeze layerTraining layerANoneConv, RNN, and FCBConv and FCRNNCConvRNN and FCWhile training, one pipeline for non-native English accent was trained and the other pipeline was used as is with pretrained weights of the original DeepSpeech2 model. The English accent pipeline was freezed by setting the learning rate 0 for all layers. The proceeding implementation details for non-native English accent pipeline.The training of model is done using ReLU activation in all of the layers. The proposed loss function as discussed in Section 4.1.1 is used as a criterion. Stochastic gradient descent (SGD) optimizer is used with dynamic learning rate, starting froml = 1 × e−3 with decay rate of 1 × e−1. The model is trained for 200 epochs. The experiments are performed on the system having Nvidia’s 1080 Ti GPU having 3584 Cuda cores and 11 GB cud memory. The system contains 16 GB DDR3 RAM and a 2.8 quad-core processor. The total dataset is split in a 70–30 ratio. A total of 70% of the data are utilized for learning the model and the remaining 30% of data are used to evaluate the learning of the model. The model’s architecture is designed and trained on paddle paddle framework. Some of the main python libraries for data manipulation and audio signal processing are used such as Pandas, Numpy, SciKit, and PyAudio.The model hyperparameters are shown in Table4.Table 4
Parameters and their values while implementation.
ParametersValueOperating systemUbuntu 18.04Frame workPaddle paddleLanguagePython 3.7CPUCore-i7 (7th gen.)RAM16 GB (DDR3)GPU1080 Ti (11 GB memory, 3584 cores)Batch size256Epochs200Drop out0.4Learning ratel = 1 × e−3Loss functionProposed Section 4.1.1OptimizerSGD


### 4.1.1. Loss Function

Most existing ASR models were optimized using some cost functions that take the predicted outputy′ and ground truth y. The difference y and y′ is used to optimize the model parameters. The proposed optimization technique intends to reduce the feature gap between the South Asian and European accents. So, the proposed loss calculation function uses the features of South Asian and European accent audio from FC2 layer, and the mean square difference of these features is used as a loss to optimize the model parameters as shown in the following equation:(6)da,e=1Na1−e12+a2−e22+⋯+ai−ei2+⋯+an−en2,here a is feature vector of South Asian accent and e is feature vector of European accent on the same transcription and N is number of samples. The objective of this loss function is to decrease the difference between the feature vector of South Asian and European accents to reduce the accuracy gap in ASR for South Asian and European accents.


## 4.1.1. Loss Function

Most existing ASR models were optimized using some cost functions that take the predicted outputy′ and ground truth y. The difference y and y′ is used to optimize the model parameters. The proposed optimization technique intends to reduce the feature gap between the South Asian and European accents. So, the proposed loss calculation function uses the features of South Asian and European accent audio from FC2 layer, and the mean square difference of these features is used as a loss to optimize the model parameters as shown in the following equation:(6)da,e=1Na1−e12+a2−e22+⋯+ai−ei2+⋯+an−en2,here a is feature vector of South Asian accent and e is feature vector of European accent on the same transcription and N is number of samples. The objective of this loss function is to decrease the difference between the feature vector of South Asian and European accents to reduce the accuracy gap in ASR for South Asian and European accents.


## 4.2. Dataset

We have used common voice [53] dataset for fine-tuning of DeepSpeech2 [54] models. common voice dataset was recorded for more than 18 languages by Mozilla for the purpose of research. It consists of total 1087 hours of audio files, from which 780 hours were validated with transcription. This dataset was recorded with both male and female voices with the ratio of 47% and 11% at a sampling frequency of 16 kHz. A detailed description of dataset accent according to the region is listed in Table 5.Table 5
Percentage of different English accents in common voice dataset.
Share (%)Accent23United States English9England English4India and south Asia (India, Pakistan, and Sri Lanka)3Canadian English3Australian English1New Zealand English1Southern African (South Africa, Zimbabwe, and Namibia)1Scottish EnglishAs stated earlier, we are focusing on non-native English accents, so we used a subset of this dataset by filtering out Pakistani, Indian, Dutch, and Sri-Lankan speaker’s recordings, with the help of Algorithm 1. Because accent variation is affected by the geographical area in which the speaker grows up and lives as well as by factors such as social class, culture, education, and working environment. All of these factors have an impact on the accuracy of the automatic speech recognition system. After splitting desired recorded files, we got a total of 10,219 audio files with an average playtime of 5 seconds. Table6 shows the further splitting of training and testing classes accordingly with respect to gender.Table 6
Total number of audio files in training and test set of non-native English accents with respect to gender.
UtterancesTraining setTest setMale4,4441,112Female3,730933Total8,1742,045


## 4.3. Evaluation Measures

The model evaluation parameters that we have selected are word error rate (WER), match error rate (MER), and word information rate (WIR).


### 4.3.1. Word Error Rate

WER is one of the most common evaluation parameters for ASR models and it provides a good comparison between the results of our proposed model and other related work done so far. The WER tells the rate of error in the transcript generated by ASR by comparing it to the original transcript. It can be calculated by the following equation:(7)W.E.R=S+I+DN,where N is the total number of words spoken in the original transcript.(8)N=S+D+H,here, S is the number of substitutions. I is the number of insertions, D is the number of deletions, and H is the total number of hits, i.e., correctly transcribed words.


### 4.3.2. Match Error Rate

Match error rate tells the probability of given input-output word matches being incorrect. It can be calculated by equation.(9)M.E.R=S+I+DH+S+I+D=1−HN.Unlike in WER, hereN is sum of all four terms.(10)N=H+S+I+D.


### 4.3.3. Word Information Loss

Word information loss gives the probability of any input word is matched with an equal output word and vice versa. It can be calculated by equation.(11)W.I.L=H2H+S+IH+S+D.All three evaluation metrics represent the errors and loss in the output, hence, the lesser the value is, the better the model predicts. However, WER is not an actual percentage as it has no upper bound limit because of the insertion I parameter. So, WER can only be used to compare different models while MER and WIL can be interpreted as how well the model performs.


## 4.3.1. Word Error Rate

WER is one of the most common evaluation parameters for ASR models and it provides a good comparison between the results of our proposed model and other related work done so far. The WER tells the rate of error in the transcript generated by ASR by comparing it to the original transcript. It can be calculated by the following equation:(7)W.E.R=S+I+DN,where N is the total number of words spoken in the original transcript.(8)N=S+D+H,here, S is the number of substitutions. I is the number of insertions, D is the number of deletions, and H is the total number of hits, i.e., correctly transcribed words.


## 4.3.2. Match Error Rate

Match error rate tells the probability of given input-output word matches being incorrect. It can be calculated by equation.(9)M.E.R=S+I+DH+S+I+D=1−HN.Unlike in WER, hereN is sum of all four terms.(10)N=H+S+I+D.


## 4.3.3. Word Information Loss

Word information loss gives the probability of any input word is matched with an equal output word and vice versa. It can be calculated by equation.(11)W.I.L=H2H+S+IH+S+D.All three evaluation metrics represent the errors and loss in the output, hence, the lesser the value is, the better the model predicts. However, WER is not an actual percentage as it has no upper bound limit because of the insertion I parameter. So, WER can only be used to compare different models while MER and WIL can be interpreted as how well the model performs.


## 4.4. Results

Before any modifications happened in DeepSpeech2, we loaded pretrained model on LibriSpeech dataset. As the majority of speakers of this dataset were from US, the model achieved WER of around 6% on the test set on US English accents. As the model was trained on American people and the test was also containing American people, that is why we obtained 6% WER. However, the same model, when evaluated using common voice dataset, gave a drastically high WER of 43% for South Asian accents. The reason for 43% WER is pretrained model of DeepSpeech2, which is not trained for non-native American or non-native English guys, that is why we use parallel pipeline for processing for the non-native English peoples. This is called a learning network. When it finds the input of non-native speakers for English it learns and updates its weights accordingly, meanwhile, we have frozen the learning rate for the freeze network, hence, we can save the performance of our model for the native English users and weights would not be changed for this pipeline. Whereas if the user is not-native English speaker then the learning network will entertain that user and update the weights. That is why our model is working better than other models due to its learning controls.The training of DeepSpeech2 is done in three different layer configurations as mentioned in Table3. The loss comparison graph of configurations A, B, and C is shown in Figure 4. The purpose of experimenting through these configurations is that we want to make DeepSpeech2 perform better for South Asian accents by transfer learning. We experimented with different dropout ratios and the best results were achieved with a dropout ratio 0.7. All the results shown here of different configurations have the same dropout ratio of 0.7.(1)
For configuration A, where all layers were learned using a common voice dataset, the model achieved W.E.R of 35.35% on the validation set. The training and validation cost of this modification is shown in Figure5(a).(2)
For configuration B, the weights of RNN layer have been learned by freezing the both CNN and FC layers. In this configuration, the model retained its low-level features learned from LibriSpeech dataset and learned only the new high-level features from the common voice dataset. The WER achieved on the validation set is 20.419%. Training and validation cost for these configurations is shown in Figure5(b).(3)
Finally, for configuration C, we learned the weights of RNN and FC layers by freezing the convolutional layer. W.E.R achieved on the validation set is 18.0859%. The cost of training and validation is shown in Figure5(c).(4)
We performed fine tuning on both the European accent and South Asian accent datasets. The results deduced after fine-tuning showed that it behaved differently for both datasets, as for European accent the W.E.R increased from 6.8% to 7.0% whereas for South Asian accent W.E.R is reduced from 51% to 18.08%.Figure 4
The image shows a calculated loss after training and validation at config A, config B, and config C.Figure 5
Training is done for 200 epochs. The cost or loss of training in different configurations is shown. (a) The trend of loss function for training and validation of the model in configuration A, (b) the trend of loss function for training and validation of model in configuration B, and (c) the minimum loss for training and validation of model in configuration C.
(a)(b)(c)The WER, MER, and WIL rate of the model on test set are shown in Table7.Table 7
Results of all three-layer configurations of the proposed model in terms of WER, MER, and WIL.
ConfigSouth Asian accentWestern accentWER (%)MER (%)WIL (%)WER (%)MER (%)WIL (%)A35.3529.6830.9810.3612.6511.22B20.41918.5419.359.688.699.85C18.085915.3615.257.06.87.2We contrasted our DeepSpeech2 algorithm to Apple Dictation, Bing Speech, Google Speech API, and wit.ai, which are all for profit speech technologies. Our test is intended to monitor success in noisy situations. This circumstance complicates the evaluation of web audio APIs: whenever the SNR is just too small or, in certain situations, whenever the phrase is too lengthy, such algorithms will provide no results. As an outcome, we limit our analysis to phrases under that all algorithms gave a not-a-void outcome. Table8 shows the outcome of assessing each system on our test files.Table 8
The comparison between nonoptimized and optimized models with respect to WER and CER.
ModelNonoptimizedOptimizedWERCERWERCERDeepSpeech215.574.52——DeepSpeech2 KENLMc10.463.6810.452.96DeepSpeech2 KENLMo10.753.7910.662.89DeepSpeech2 and KENLM (c + O)9.93.619.912.80The comparison of the proposed model and DeepSpeech2 model is shown in Table9.Table 9
Comparison of DeepSpeech2 and proposed model’s WER on English and non-native English accent.
ModelWER (English accent) (%)WER (non-native English accent) (%)DeepSpeech 2643Proposed718.08


## 5. Conclusion

ASR is being used extensively to enable natural human-machine interaction, but not pliable enough for South Asian accent for English language for which almost 200 million people are unable to use its applications. Our contribution towards resolving this obstacle is the proposed system, that is, inspired by DeepSpeech2. The proposed method provides the two pipelined deep learning architectures that achieve minimum character error rate (CER) and word error rate (WER) on common voice (CV) benchmark. By setting up different experimental configurations and modifications, we are successful in achieving minimum WER, that is, reduced from 43% to 18.08% at a lower validation cost. As this work focused on South Asian’s English accents so, there is a little bit of increase in WER and CER for English speakers. The system will be further scalable towards targeting other South Asian languages like Bengali, Urdu, Hindi, and others with more robust datasets and higher accuracy and the training of both pipelines parallelly.



---

*Source: 6825555-2022-10-04.xml*