Fast offline transformer‐based end‐to‐end automatic speech recognition for real‐world applications

ETRI JournalEarly View ORIGINAL ARTICLEFree Access Fast offline transformer-based end-to-end automatic speech recognition for real-world applications Yoo Rhee Oh, Yoo Rhee Oh orcid.org/0000-0002-1557-0538 Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Republic of KoreaSearch for more papers by this authorKiyoung Park, Corresponding Author Kiyoung Park pkyoung@etri.re.kr Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Republic of Korea Correspondence Kiyoung Park, Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Republic of Korea. Email: pkyoung@etri.re.krSearch for more papers by this authorJeon Gue Park, Jeon Gue Park Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Republic of KoreaSearch for more papers by this author Yoo Rhee Oh, Yoo Rhee Oh orcid.org/0000-0002-1557-0538 Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Republic of KoreaSearch for more papers by this authorKiyoung Park, Corresponding Author Kiyoung Park pkyoung@etri.re.kr Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Republic of Korea Correspondence Kiyoung Park, Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Republic of Korea. Email: pkyoung@etri.re.krSearch for more papers by this authorJeon Gue Park, Jeon Gue Park Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Republic of KoreaSearch for more papers by this author First published: 08 December 2021 https://doi.org/10.4218/etrij.2021-0106AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract With the recent advances in technology, automatic speech recognition (ASR) has been widely used in real-world applications. The efficiency of converting large amounts of speech into text accurately with limited resources has become more vital than ever. In this study, we propose a method to rapidly recognize a large speech database via a transformer-based end-to-end model. Transformers have improved the state-of-the-art performance in many fields. However, they are not easy to use for long sequences. In this study, various techniques to accelerate the recognition of real-world speeches are proposed and tested, including decoding via multiple-utterance-batched beam search, detecting end of speech based on a connectionist temporal classification (CTC), restricting the CTC-prefix score, and splitting long speeches into short segments. Experiments are conducted with the Librispeech dataset and the real-world Korean ASR tasks to verify the proposed methods. From the experiments, the proposed system can convert 8 h of speeches spoken at real-world meetings into text in less than 3 min with a 10.73% character error rate, which is 27.1% relatively lower than that of conventional systems. 1 INTRODUCTION Owing to the rapid advances in automatic speech recognition (ASR) research [1-4], they have been widely adopted in many practical applications, such as automatic response systems, automatic subtitle generation, and meeting transcription. Deep neural network (DNN) incorporated with the traditional hidden Markov Model (HMM) has worked powerfully and increased speech recognition accuracy to the commercial level. A recent breakthrough in deep learning (DL) from natural language processing solved a sequence generation problem using a novel strategy [5, 6]. Further, a transformer model in which sequences are minutely processed with self-attention achieves state-of-the-art performances in many tasks. In ASR, where the speech recognition problem is formulated as the generation of the most probable sequence of words given a spoken utterance as an input sequence, a transformer-based end-to-end speech recognition technique has achieved state-of-the-art performance and a breakthrough because DL technology was used with HMM [7, 8]. However, it is well-known that transformer is insufficient at processing long sequences in terms of performance and computation speed, and speech signals are usually represented by much longer sequences than the written sentences. There have been many research studies to overcome this drawback, including [9-11]. There is an abundance of speeches already recorded and published that need to be transcribed, and more data are being generated daily [12]. Hence, it is crucial to recognize large speech databases very quickly at the lowest cost. In this study, we present efficient methods to use transformer in continuously spoken long speech recognition. 1.1 Literature review In 2017, a DL model, transformer, was introduced for machine translation tasks [8]. Transformer has been adopted in many natural language processing tasks and has produced significant performance improvements. In 2018, transformer was adopted for speech recognition tasks with significant improvements compared with recurrent neural network (RNN)-based speech recognition [7, 13, 14]. Accordingly, there have been many studies on transformer-based end-to-end speech recognition, which achieved state-of-the-art speech recognition [15-17]. Moreover, Karita [18] successfully integrated connectionist temporal classification (CTC) into transformer-based end-to-end speech recognition for rapid training and subsequently improved the speech recognition performance. Recently, the joint CTC/transformer approach have been used in various research studies [10, 16, 19-32]. Therefore, we focus on CTC/transformer-based end-to-end speech recognition. To increase the throughput of RNN-based speech recognizers, Amodei et al., Braun et al., and Oh et al. [33-35] used several batch processing approaches. In particular, Braun et al. and Oh et al. [34, 35] aimed to accelerate GPU parallelization. Seki et al. [36, 37] proposed a multiple-utterance multiple-hypothesis vectorized beam search in CTC-attention-based end-to-end speech recognition using a VGG-RNN-based encoder–decoder and showed the decoding throughput increased using a GPU. Motivated by Seki et al. [36, 37], we aim to successfully adopt the vectorized beam search in CTC/transformer-based end-to-end speech recognition for rapid and efficient decoding of a large speech database. The CTC-prefix decoding used in CTC/transformer end-to-end speech recognition requires computation over an entire input sequence for each iteration [38] and is not parallel processing but sequential processing, which hinders the GPU parallelization. Because this is a bottleneck in fast processing, several time-restricted CTC-prefix decoding methods [36, 39] have been proposed. Seki et al. [36] proposed a windowing method using attention weights of a VGG-RNN decoder network with the assumption that the time alignment of the CTC-prefix decoding is similar to that of the attention decoding. Miao et al. [39] proposed truncated CTC-prefix decoding using the previous and current CTC probabilities of a blank symbol at each iteration. For fast decoding, we propose time-restricted CTC-prefix decoding using a different method. Voice activity detection (VAD) is one solution for decoding long utterances robustly with low resources in an end-to-end speech recognizer [40]. Yoshimura et al. [40] proposed a CTC-based VAD integrated into an end-to-end speech recognizer and compared it with an energy-based VAD method and a time-delay neural network (TDNN)-based VAD. In this study, we propose a DNN-based VAD using a different method for the vectorized beam search in CTC/transformer end-to-end speech recognition. 1.2 Proposed method and its contribution In this study, we propose a method to recognize a large speech database rapidly and cost-effectively for CTC/transformer-based end-to-end speech recognition. To this end, we adopt the multiple-utterance multiple-hypothesis vectorized beam search decoding of Seki et al. and Miao et al. [36, 39] in transformer-based speech recognition. We observed that the baseline end-of-speech detection method fails if an abnormal utterance is encountered because the method is based on the absolute value of CTC probability. Thus, we propose an end-of-speech detection method using CTC-based time information. To reduce the deterioration of GPU parallelization, CTC-prefix decoding is moved from the GPU to the CPU. To unburden the CPU load, we propose a time-restricted CTC-prefix decoding method using the previous CTC probabilities of the target symbol and its blank symbol for each iteration because it is difficult to determine the time information from a transformer decoder, unlike [36, 39]. Another issue is the limited GPU memory size because batched utterances are padded with the maximum utterance length before the vectorized beam search [36, 37]. Therefore, we propose a method to segment long speeches into smaller pieces for batch decoding and create a batch having speech segments of similar lengths. The contributions of this study are summarized as follows. Multiple-utterance multiple-hypothesis batched beam search for transformer As a prior work, multiple-utterance multiple-hypothesis batched decoding was employed for a joint CTC/attention-based VGG-RNN. Motivated by this prior work, we extend the multiple-utterance multiple-hypothesis batched decoding for transformer-based speech recognition. Time-restricted CTC-prefix scoring for transformer Influenced by time-restricted CTC-prefix scoring for a VGG-RNN-based model, we present time-restricted CTC-prefix scoring using CTC-based time information instead of attention weights. Improvement in end-of-speech detection We propose an improved end-of-speech detection based on estimated time information. Conventional end-of-speech detection works well in matched conditions with speeches in the training corpus, but if an abnormal speech is encountered, the method could fail to detect end of speech. Thus, the decoding efficiency is degraded. To solve this problem, we improve conventional end-of-speech detection. Utterance segmentation for efficient decoding with the transformer To efficiently manipulate GPU-based batched beam search decoding for the transformer, it is crucial to split utterances to be decoded. First, we adopted a DNN-VAD-based segmentation to split at pauses. Using this method, the decoding speed could be significantly accelerated with little accuracy degradation. Then, it was shown that hard segmentation could also be used as a splitting method with minor accuracy degradation. It is also analyzed that sorting and making batches of utterances of similar lengths speed up the decoding. By integrating these techniques, we could deploy highly efficient end-to-end speech recognizers for real-world applications. The rest of this paper is organized as follows. Section 2 reviews a CTC/transformer-based speech recognition. In Section 3, the methods employed to speed up the recognition for large speech corpus are presented. In Section 4, the performance of the proposed speech recognition is evaluated with a public dataset and a real-world meeting corpus of multiple speakers. Finally, we conclude this study with our findings in Section 5. 2 CTC/TRANSFORMER-BASED SPEECH RECOGNITION Vaswani et al. [8] proposed a sequence-to-sequence attention-based encoder–decoder network, transformer, for a neural machine translation task. With an outstanding performance, transformer is successfully adopted in speech recognition [7, 13, 16], and this work adopts the transformer network of [16]. As shown in Figure 1, the encoder network comprises subsampling, input-embedding, position-encoding, Ne encoder blocks, and normalization layers. The decoder network comprises output-embedding, positional-encoding, Nd-decoder blocks, normalization, linear, and softmax layers. A linear layer and a softmax layer are also used for a CTC decoder, which is shown in the orange line in the figure. Each encoder block comprises multihead attention and feed-forward layers. Each decoder block comprises masked multihead attention, multihead attention, and feed-forward layers. Layer normalization is applied before each multihead attention and feed-forward, and residual addition is applied after each of them. The following subsections briefly explain a joint CTC/transformer score and a beam search decoding for speech recognition inference. FIGURE 1Open in figure viewerPowerPoint Overall structure of the connectionist temporal classification (CTC)/transformer-based speech recognizer used in this work [16]. Encoder and decoder blocks are depicted in blue and green lines, respectively. A CTC decoder is shown in an orange line 2.1 Joint CTC/transformer score In Karita and Seki et al. [18, 36], a joint CTC/attention decoding is performed to maximize the logarithmic linear combination of the sequence probabilities obtained by the transformer and CTC decoders. For the l-length sequence, y1 : l, of a speech representation, h, this framework calculates a joint CTC/transformer score [18] as follows: (1)where and are the CTC-prefix and attention probabilities, respectively, and λ is the CTC weight. The attention probability, , is calculated as follows: (2)where pa(yl|y1 : l − 1, h) is the attention output probability at time l given y1 : l − 1 and h, as shown in Figure 1. Next, CTC [38] maps a frame-wise speech feature vector to a token-wise output sequence monotonically. Let h be a frame-wise T-length speech representation and y1 : L be a token-wise L-length output sequence, , with a token set C. The CTC introduces a frame-wise T-length output sequence, , with an extended token set . zt can be defined as follows: (3) (4) (5) (6)where the first and last nonblank tokens are y1 and yL, respectively; is the previous nonblank token up to t. The best output sequence of Z is obtained by finding a sequence that maximizes a CTC-prefix probability [36, 39, 41]. The CTC-prefix probability accumulates the probabilities of all possible sequences with a prefix sequence. That is, for an l-length hypothesis y1 : l of a speech representation h, the CTC-prefix probability, , is calculated as follows: (7)where is the CTC-forward probability up to time t − 1 for y1 : l − 1; is the CTC-posterior probability at time t given y1 : l − 1 and h (Figure 1). 2.2 Beam search decoding In this work, we use a beam search decoding [42] to find the output sequence for an input utterance. The beam search is a beam-pruned breadth-first search, as summarized in Algorithm 1. Let x be a speech feature. Using a transformer encoder, x is converted into the intermediate representation h, where is the length of h. Then, h is converted into the text sequence y by performing the B-width beam search, where (1) is used as the scoring function to evaluate the reliability of a hypothesis (Line 3 of Algorithm 1). At the l-th step of the beam search, the l-length hypothesis scores, , are calculated using h, Yl − 1, and C (Line 7 of Algorithm 1). Notably, the notation A[B] means that a matrix A has the shape of B. I[B] is obtained by selecting the indices of the B highest scores of (Line 10 of Algorithm 1). Then, the l-length hypothesis set, , is obtained by selecting the l-length reliable hypotheses using Yl − 1, C, and I[B] (Line 11 of Algorithm 1). Notably, a ∘ B indicates that a is concatenated with each item of B. Next, is updated if <eos>-ended hypothesis is confident (Lines 12 to 15 of Algorithm 1). The beam search is terminated if the decoding step l is greater than (Line 4 of Algorithm 1) or if end of speech is detected (Lines 17 to 19 of Algorithm 1). Finally, the output sequence is obtained with the most probable hypothesis in (Line 21 of Algorithm 1). 3 RAPID AND EFFICIENT TRANSFORMER-BASED SPEECH RECOGNITION 3.1 Rapid decoding based on batched beam search GPU parallelization can be accelerated by batch processing [33-37]. As in Seki et al. [36, 37], we adopt a multiple-utterance multiple-hypothesis batched beam search for a transformer-based end-to-end speech recognizer to efficiently parallelize speech recognition for multiple utterances. The batched beam search is briefly described as follows: the detailed explanation is in Seki et al. [36]. Let be the speech features obtained from U utterances. X is batched as , where xi is length-extended from xi. The length is , and the extended values are padded with zeros. Using a transformer encoder, is converted into the intermediate representations , where each of them can be converted in parallel. Then, is decoded into the text sequence set by performing a B-width batched beam search. For the scoring function, (1) is used to evaluate each hypothesis of in parallel (Line 3 of Algorithm 2). At the l-th step of the batched beam search, the score set, , of B l-length hypotheses of U speech features is calculated using , and C, where each score can be calculated in parallel (Line 6 of Algorithm 2). is obtained by selecting the indices of the B highest scores of the U speech feature using (Line 9 of Algorithm 2). Then, the l-length hypothesis set, , is obtained by selecting the B l-length reliable hypotheses for each speech feature using , and (Line 10 of Algorithm 2). Next, confident <eos>-ended hypotheses for each utterance are added into (Lines 11 to 18 of Algorithm 2). The beam search is terminated if the decoding step l is greater than (Line 4 of Algorithm 2) or if end of speech is detected for all utterances (Lines 19 to 22 of Algorithm 2). Finally, the output sequence is obtained with the most reliable hypothesis in for each utterance (Line 24 of Algorithm 2). When compared with [36, 37], this work uses a different scoring function for a CTC/transformer-based speech recognition (Line 3 of Algorithm 2) and improves an end-of-speech detection based on CTC (Line 19 of Algorithm 2). For a rapid decoding, we also propose a time-restricted CTC scoring of the scoring function. 3.2 CTC-based end-of-speech detection and time-restricted CTC-prefix scoring In this section, we first explain a baseline end-of-speech detection [43] and then propose rapid decoding methods based on a CTC-prefix score. 3.2.1 Baseline end-of-speech detection As mentioned in Section 3.1, the beam search decoding finishes when a confident <eos>-ended hypothesis is detected for a rapid completion. Based on Watanabe et al. [43], the baseline end-of-speech detection detects end of speech if the following (8) is satisfied: (8)In this work, the predefined parameters M and Dend are set to 3 and e−10, respectively, as in Watanabe et al. [43]. 3.2.2 Proposed CTC-based end-of-speech detection The baseline end-of-speech detection of [43] performs well in a normal condition. However, the absolute probability values of hypotheses are unreliable in a mismatched condition, between training and decoding, which can occur in real-world applications. To prevent such a malfunction that slows down speech recognition, we propose a CTC-based end-of-speech detection method that uses the estimated time of the last label (<eos>) of each completed hypothesis. Based on a CTC-prefix score, for the j-th l-length hypothesis, , of an i-th feature representation, the start time index and end time index of the last label, , are estimated as follows: (9) (10)where is a text sequence that ends with a blank symbol concatenated after y. That is, is a time index that maximizes the CTC-forward probability for within a specified time-index range. Similarly, is one for . As shown in Figure 2A, we validate the estimated time index by obtaining the start time of the decoded output text for a Korean utterance using the proposed start time index. Notably, the upper and lower parts show the waveform in yellow and its spectrum in gray for the utterance. The estimated start time is shown in red. The start time is nearly estimated at the start position of each syllable. Moreover, Figure 2B shows an example of the start time indices for a wrong hypothesis having repetition errors, which errors can occur in attention-based speech recognition. As shown in the figure, the repetition errors are , and the start time indices for the errors are the same as lenh = 100 due to the limited search range of τ. FIGURE 2Open in figure viewerPowerPoint Example of time information for each decoded output unit using estimated start time index Based on the property depicted in Figure 2B, the CTC-based end-of-speech detection method is proposed, as in Algorithm 3. If the baseline end-of-speech detection method fails to predict end of speech, the start time index is examined for each hypothesis in the confident <eos>-ended hypothesis set for hi. If the start time index is equal to lenh more than C times, end of speech is detected. In this study, the parameter C is set to 2. 3.2.3 Proposed time-restricted CTC-prefix score for rapid decoding Because the sequential calculation of a CTC-prefix score limits parallel processing, we propose a CPU-based and time-restricted CTC-prefix score for a joint CTC/attention-based transformer to improve computational efficiency. Calculation of the CTC-prefix score of (7) is restricted within the proposed time range, as in Seki et al. [36]. The proposed time-restricted CTC-prefix score is defined as follows: (11)where and indicate the start and end times to be calculated, respectively. These are calculated as follows: (12) (13)where M1 and M2 are tunable margin parameters. For the batch processing of , the restricted range is defined as follows: (14) (15)The time-restricted CTC-prefix score can be computed in a small amount of the calculation. For a small but frequent calculation in a GPU device, memory transfer load between CPU and GPU can be a bottleneck [35]. Therefore, the calculation of the CTC-prefix score is moved from the GPU device to a CPU device to reduce the GPU memory transfer load. The proposed time-restricted CTC-prefix score reduces the amount of computation based on the proposed estimated time information because a transformer decoder hardly captures time information. By contrast, Seki et al. [36] restricted the CTC-prefix score using time information from an RNN-based attention decoder. Miao et al. [39] only limited the end time based on a differently estimated end time. 3.3 Offline recognition of long utterances When it comes to the recognition of long utterances, the performance of transformer-based end-to-end speech recognition tends to degrade significantly due to the characteristics of the self-attention of the transformer and sensitivity to the utterance length of training data [44]. Meanwhile, the computational complexity for a self-attention quadratically increases in proportion to the utterance length [8, 45]. To tackle these issues, we perform segmentation before recognition. Two segmentation methods are tested: The first is a split at a short pause with a VAD, and the second is simple hard segmentation. In Yoshimura et al. [40], VAD information is used as a triggering sign for the decoder in real time. In this work, explicit segmentation is performed beforehand using a DNN-based VAD. In conventional DNN-HMM ASR systems coupled with language models, the recognition results are searched by back-tracking the optimal path using the Viterbi algorithm. Usually, the back-tracking point is determined as an end of an utterance. However, for long speeches, finding the relevant short pauses and using them as back-tracking points make it possible to generate ASR results rapidly and hence to reduce the user-perceived latency. Finding short pauses or detecting voice activity in continuous speech is a difficult task and one of the oldest topics in speech-related researches. Traditionally, the methods to track the energies of speech and nonspeech are commonly used. These methods suffer when the signal-to-noise ratio is dynamically changes. In our previous work [35], the outputs of acoustic model (AM) were successfully employed to detect voice activity in DNN-HMM-based ASR systems. In DNN-HMM-based ASR systems, a DNN is trained so that each output node estimates a posteriori probability of a state given an input feature vector xt at time t, P(sk|xt), where a state sk corresponds to the part of a phoneme. The probabilities were directly used as the AM scores, which were combined with language model scores to find optimal ASR results. In a previous work [35], these probabilities were successfully used to compute probabilities of voice presence in input signal as follows. Let be the output value of the i-th node of the neural network at time t. Then, speech presence probability () and speech absence probability () at time t are estimated as follows: (16) (17) (18)where and are output nodes corresponding to the speech and noise states, respectively. A frame is regarded as a nonspeech frame if the log-likelihood ratio is larger than a given threshold. To obtain a robust decision, the frame-wise results are smoothened over multiple frames, as in Oh et al. [35]. We reused the DNN-based VAD algorithm to segment long speeches for transformer-based end-to-end speech recognition system. Segmentation based on DNN-VAD produces good performance in splitting long utterances in general; however, this requires significant computation resources. In DNN-HMM-based ASR systems, these probabilities are computed for ASR, and there are no extra computations for VAD. However, for the transformer-based end-to-end speech recognition system, there is no such component as AM, and this computation is only for the VAD, which requires heavy computation with GPU for large DNN models. It is also possible to use the simpler VAD algorithm, as in [46]. In this work, we attempted hard segmentation by segment length, that is, splitting long utterances into short pieces of the same length while not considering truncation. Hard segmentation requires no computation, and the resulting segments are of the same length. This reinforces the merits of the batch processing in Section 3.1. 4 EXPERIMENTS 4.1 Baseline ASR system and speech corpus 4.1.1 Transformer-based end-to-end model The proposed transformer model of the end-to-end speech recognizer was trained using ESPnet, an end-to-end speech processing toolkit [47], and the decoding was performed using a modified version of the toolkit. During the decoding, an attention cache was applied in all experiments for acceleration [48]. As an input feature, 80-dimensional log-Mel filterbank coefficients were extracted for every 10-ms analysis frame. The pitch feature was not used because, in the Korean language, tone and pitch are not important features. During training and decoding, a global cepstral mean and variance normalization was applied to the feature vectors. Most hyperparameters for the transformer model inherit values from the default settings of the toolkit. The encoder comprises two convolutional layers, a linear projection layer and a positional-encoding layer, followed by 12 self-attention blocks with layer normalization. An additional linear layer is used for CTC. The decoder has six self-attention and encoder–decoder attention blocks. For every Transformer layer, we used 2048-dimensional feed-forward networks and four attention heads with 256 dimensions. The training was performed using the Noam optimizer [8], warmup steps, label smoothing, gradient clipping, and accumulating gradients [49]. One hundred epochs of training were performed without early stopping. The text tokenization was performed in terms of character units. In Korean, a character comprises two or three graphemes and corresponds to a syllable. Although the number of plausible characters in Korean is greater than 10 000, only 2273 tokens are used as output symbols, including digits, alphabets, and a spacing symbol, which are seen in a training corpus. No other text processing was performed, except for the removal of punctuation marks and tokenization. 4.1.2 Speech corpus To train the end-to-end ASR model, we used approximately 12.4k h of Korean speech corpus, which comprises various sources, including read digits and sentences, recordings of spontaneous conversation, and mostly (about 11.1k h) broadcast data [50]. Therefore, the training data contain various utterance lengths. All sentences were manually transcribed, and long sentences were automatically segmented using the transcription and aligning process. Utterances longer than 30 s were excluded during training. As for the test corpus, meetings at a public office were recorded. An automatic meeting transcription system was introduced for evaluation. In total, 8 h of recordings were made from seven meetings. In each meeting, 4 to 22 people participated, and every participant had his or her gooseneck-type microphone. Each microphone was turned on only while the corresponding speaker uttered, and this was recorded separately. However, there were some overlaps and crosstalk from adjacent speakers, which were ignored in the manual transcript