FastCorrect: Fast Error Correction with Edit Alignment for Automatic Speech Recognition

1 Introduction . In recent years , error correction techniques [ 1 , 5 , 22 , 26 , 33 ] have been widely adopted to refine the output sentences from an ASR model for further WER reduction . Error correction , a typical sequence to sequence task , takes the sentence generated by an ASR model as the source sequence and the ground-truth sentence as the target sequence , and aims to correct the errors in the source sequence . Previous works on ASR error correction [ 22 , 26 ] usually adopt an encoder-decoder based autoregressive generation model . While achieving good WER reduction , autoregressive models suffer from slow inference speed , and do not satisfy the latency requirements for online ASR services . For ∗This work was conducted at Microsoft . Corresponding author : Xu Tan , xuta @ microsoft.com 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . example , the latency of our internal product ASR system is about 500ms for an utterance on a single CPU , but the latency of the autoregressive correction model alone is about 660ms , which is even larger than the original ASR system and unaffordable for online deployment . Non-autoregressive ( NAR ) models can speed up sequence generation by generating a target sequence in parallel , and attract much research attention , especially in neural machine translation ( NMT ) [ 6 , 8 , 9 , 41 ] . Unfortunately , direct application of NAR models designed for NMT to ASR error correction leads to poor performance . According to our experiments , using a popular NAR model from NMT [ 9 ] for error correction even increases WER , i.e. , the correction output is worse than the original ASR output . Different from NMT where almost all input tokens need to be modified ( i.e. , translated to another language ) , the modifications in ASR correction are much fewer but more difficult . For example , if an ASR model achieves 10 % WER , only about 10 % input tokens of the correction model need to be modified , and these tokens are usually difficult to identify and correct since they have already been mistaken by the ASR model . Thus , we need to take the characteristics of ASR outputs into consideration and carefully design NAR models for ASR error correction . In ASR error correction , the source and target tokens are aligned monotonically ( unlike shuffle error in neural machine translation ) , and ASR accuracy is usually measured by WER based on the edit distance . Edit distance provides the edit and alignment information such as insertion , deletion and substitution on the source sentence ( the output of an ASR model ) in order to match the target ( ground-truth ) sentence , which can serve as precise guidance for the NAR correction model . Based on these observations , in this paper , we propose FastCorrect , a novel NAR error correction model that leverages and benefits from edit alignment : • In training , FastCorrect first obtains the operation path ( including insertion , deletion and substitution ) through which the source sentence can be modified to target sentence by calculating the edit distance , and then extracts the token-level alignment that indicates how many target tokens correspond to each source token after the insertion , deletion and substitution operations ( i.e. , 0 means deletion , 1 means unchanged or substitution , ≥2 means insertion ) . The token-level alignments ( token numbers corresponding to each source token ) are used to train a length predictor and to adjust the source tokens to match the length of the target sentence for parallel generation . • In inference , we can not get token alignments as the ground-truth sentence is not available . We use the length predictor to predict the target token number for each source token and use the predicted number to adjust the source tokens , which are then fed to the decoder for target sequence generation . With this precise edit alignment , FastCorrect can correct the ASR errors more effectively , using a length predictor to locate which source token needs to be edited/corrected and how many tokens will be corrected to , and then using a decoder to correct the tokens correspondingly . Since current ASR models have already achieved high accuracy , there might be not many errors in ASR outputs to train a correction model , even if we have large-scale datasets for ASR model training . To overcome this limitation , we use the crawled text data to construct a pseudo correction dataset by randomly deleting , inserting and substituting words in the text data . When substituting word , we use a homophone dictionary considering that ASR substitution errors are mostly from homophones . Those randomly edited sentences and their original sentences compose the pseudo sentence pairs for correction model training . In this way , we first pre-train FastCorrect on the large-scale pseudo correction dataset and then fine-tune the pre-trained model on the limited ASR correction dataset . The contributions of this work are as follows : • To our knowledge , we are the first to propose NAR error correction for ASR , which greatly reduces the inference latency ( up to 9× ) compared with its autoregressive counterpart while achieving nearly comparable accuracy . Our method also outperforms the popular NAR models adopted in machine translation and text edition by a large margin . • Inspired by the distinctive error patterns and correction operations ( i.e. , insertion , deletion and substitution ) in ASR , we leverage edit alignments between the output text from ASR models and the ground-truth text to guide the training of NAR error correction , which is critical to FastCorrect . The code of FastCorrect is available at https : //github.com/microsoft/NeuralSpeech/tree/ master/FastCorrect . 2 Background . Error Correction In the field of natural language processing , error correction aims to correct the errors in the generated sentence by another system , such as automatic speech recognition [ 26 , 31 , 33 , 11 ] , neural machine translation [ 36 ] and optical character recognition [ 27 ] . The error correction models for ASR can be divided into two categories based on whether the model can be trained in an end-to-end manner . Based on the method of statistic machine translation , Cucu et al . [ 4 ] performed error correction for ASR system . D ’ Haro and Banchs [ 5 ] proposed to use a phrase-based machine translation system to serve as a correction model for ASR . Anantaram et al . [ 1 ] used a four-step method to repair the ASR model output by ontology learning . With the increasing of training corpus , end-to-end correction models are more accurate and become popular . A language model was trained for ASR correction in Tanaka et al . [ 39 ] , which could exploit the long-term context and choose better results among different ASR output candidates . Mani et al . [ 26 ] utilized a Transformer-based model to train an ASR correction model in an autoregressive manner . Liao et al . [ 22 ] further incorporated the MASS [ 35 ] pre-training into ASR correction . Deliberation network [ 15 , 16 ] applied error correction using both the first-pass transcripts as well as the input speech . However , all the end-to-end correction models are autoregressive and unsuitable for online deployment due to large latency , hindering the industrial application of ASR correction . Considering that there does not exit shuffle error in ASR correction , we propose a novel method to align the source sentences and target sentences towards global optimum based on edit distance , which can not only keep the matched tokens in alignment as many as possible , but also detect the substitution , deletion and insertion errors during alignment . Non-autoregressive Models NAR generation , which aims to speed up the inference of autoregressive model while achieving minimal accuracy drop , has been a popular research topic in recent years [ 13 , 23 , 30 ] . Gu et al . [ 8 ] , Ma et al . [ 24 ] , Shu et al . [ 34 ] approached this problem with a set of latent variables . Shao et al . [ 32 ] , Ghazvininejad et al . [ 7 ] , Li et al . [ 21 ] developed other alternative loss functions to help the model capture target-size sequential dependencies . Wang et al . [ 41 ] , Stern et al . [ 38 ] proposed partially parallel decoding to output multiple tokens at each decoding step . Stern et al . [ 37 ] , Gu et al . [ 9 ] proposed to generate target tokens in a tree-based manner , and used dynamic insertion/deletion to iteratively refine the generated sequences based on previous predictions . FELIX [ 25 ] performed NAR text edition by aligning source sentence with target sentence greedily . However , as shown in section 5.1 , directly using existing non-autoregressive models such as Gu et al . [ 9 ] and Mallinson et al . [ 25 ] can not get satisfying results on ASR error correction . In this paper , based on the characteristics of ASR recognized text , we develop FastCorrect , which builds edit alignment between the source sentences and target sentences to guide the error correction . 3 FastCorrect . FastCorrect leverages NAR generation with edit alignment to speed up the inference of the autoregressive correction model . In FastCorrect , we first calculate the edit distance between the recognized text ( source sentence ) and the ground-truth text ( target sentence ) . By analyzing the insertion , deletion and substitution operation in the edit distance , we can obtain the number of target tokens that correspond to each source token after edition ( i.e. , 0 means deletion , 1 means unchanged or substitution , ≥2 means insertion ) . FastCorrect adopts an NAR encoder-decoder structure with a length predictor to bridge the length mismatch between the encoder ( source sentence ) and decoder ( target sentence ) . The obtained number of target tokens is used to train the length predictor to predict the length of each source token after correction , and to adjust each source token , where the adjusted source tokens are fed into the decoder for parallel generation . In the following subsections , we introduce the edit alignment , model structure and pre-training strategy in FastCorrect . 3.1 Edit Alignment . As shown in Figure 1 , the edit alignment between each token in source and target sentences can be obtained through two steps : calculating the edit paths with minimum edit distance ( the left and middle sub-figures ) , and choosing edit alignment with the highest n-gram frequency ( the right sub-figure ) . In the next , we introduce each step in detail . Calculating Edit Path Edit distance measures the dissimilarity of two sentences by counting the minimum number of edit operations required to transform the source sentence into the target sentence2 . The valid edit operations include token insertion , token deletion and token substitution . Given source sentence S = ( s1 , s2 , ... , sM ) and target sentence T = ( t1 , t2 , ... , tN ) , where M and N are the lengths of source and target sentences , the edit distance between S and T can be obtained by calculating the edit distance of prefix sentences recursively . The procedure is as follows : D ( i , j ) = min ( D ( i− 1 , j ) + 1 , D ( i , j − 1 ) + 1 , D ( i− 1 , j − 1 ) + 1 ( si 6= tj ) ) . ( 1 ) In the above equation , D ( i , j ) is the edit distance of source prefix sentence ( s1 , s2 , ... , si ) and target prefix sentence ( t1 , t2 , ... , tj ) , 1 ( · ) is the indicator function whose output is 1 when the condition is true otherwise 0 . The boundary condition of D ( i , j ) is D ( i , 0 ) = i , D ( 0 , j ) = j . The leftmost part of Figure 1 shows an example of the alignment between source sentence “ B B D E F ” and target sentence “ A B C D F ” . By recursively calculating the edit distance , we can obtain all the possible edit paths that have the minimum edit distance . In this case , we have 3 possible paths with minimum edit distance 3 , as path a , b and c shown in the middle part of this figure . Choosing Edit Alignment We introduce how to choose the edit alignment between each token in source sentence and target sentence based on the edit paths . First , for the edit paths obtained from the above procedure , we calculate the match score of each path and only keep the paths with the highest match score . We define the match score of an edit path as the number of tokens that is not changed in this path . If an edit path has a higher match score , this path is more preferred since most tokens in the source sentence can be kept . As shown in the middle part of Figure 1 , the match scores of the path a and b are both 3 because they both have 3 unchanged tokens : “ B ” , “ D ” and “ F ” , and the match score of path c is only 2 , which will be filtered3 . 2Standard edit distance usually transforms the target to the source . We change the order to transform the source to target to align with our scenario . 3Intuitively speaking , path c does not make sense because it first changes token “ D ” to “ C ” and then changes “ E ” to “ D ” . It is apparent that “ D ” should not be changed . Second , we get the edit alignment set E ( where e ∈ E represents a possible edit alignment for all tokens between the source and target sentences ) from all the edit paths obtained by now by the following rules : 1 ) For a deletion , the source token is aligned with an empty target token Ø . 2 ) For a substitution or identity , we align the source token with the substituted token or unchanged token in the target sentence , respectively . 3 ) For an insertion , the target token has no source token to align with ( e.g. , token “ C ” in path b ) , and then the target token will be aligned with its corresponding left or right source token , resulting in different edit alignments ( e.g. , path b can generate two alignments : b1 and b2 , by aligning target token “ C ” to source token “ B ” ( left ) or “ D ” ( right ) , respectively ) . Third , we select the final edit alignment e from the edit alignment set E obtained in the second step , according to the frequency of n-gram tokens of the aligned target tokens . We first build an n-gram frequency table G that contains the number of occurrences of every n-gram term in the training corpus , and then calculate the frequency score Freqscore ( e ) for each edit alignment e ∈ E on a source sentence S : Freqscore ( e ) = M∑ i=1 Freq ( e [ si ] ) ; Freq ( x ) = { G [ x ] , len ( x ) > 1 0 , len ( x ) ≤ 1 , ( 2 ) where e [ si ] represents the target tokens aligned with the source token si under alignment e , M is the number of tokens in source sentence , len ( x ) is the number of words in x and G [ x ] returns the frequency of x in the n-gram table4 . The frequency of all 1-gram is set to 0 because we are interested in differentiating token combinations . We choose the alignment e ∈ E with the largest frequency score as the final edit alignment between each token in the source and target sentences . Doing so , we encourage the edit alignment that aligns the source token with more frequent n-gram target tokens . Taking the alignment a in the rightmost part of Figure 1 as an example , where only the first source token “ B ” is aligned with more than 1 target tokens ( i.e. , “ AB ” ) . So the frequency score of alignment a is equal to the frequency of “ AB ” in n-gram table G ( i.e. , 90 ) , which is larger than that of n-gram token “ BC ” in alignment b1 and “ CD ” in alignment b2 . As a result , the alignment a is selected as the final edit alignment and the length of target tokens aligned with each source token is “ 2 1 1 0 1 ” , respectively .

Summary: In this work the authors consider the problem of correcting the transcripts produced by the ASR system in the first pass. In previous works, this has been explored with attention-based encoder-decoder models which must process hypotheses sequentially left-to-right by feeding each output token. However, instead the authors propose to investigate non-autoregressive models where all outputs can be processed in parallel. This is achieved by first predicting the number of output characters corresponding to each source character and then simply modifying the input source sequence by repeating/deleting tokens as appropriate. The main benefit of the proposed approach is that it allows for significant savings in latency during inference.