FastCorrect: Fast Error Correction with Edit Alignment for Automatic Speech Recognition
Authors: Yichong Leng
Xu Tan
Linchen Zhu
Jin Xu
Renqian Luo
Linquan Liu
Tao Qin
Xiangyang Li
Edward Lin
Tie-Yan Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Automatic speech recognition
Error correction
Non-autoregressive model
Post-processing
Edit alignment

Abstract: Error correction techniques have been used to refine the output sentences from automatic speech recognition (ASR) models and achieve a lower word error rate (WER) than original ASR outputs. Previous works usually use a sequence-to-sequence model to correct an ASR output sentence autoregressively, which causes large latency and cannot be deployed in online ASR services. A straightforward solution to reduce latency, inspired by non-autoregressive (NAR) neural machine translation, is to use an NAR sequence generation model for ASR error correction, which, however, comes at the cost of significantly increased ASR error rate. In this paper, observing distinctive error patterns and correction operations (i.e., insertion, deletion, and substitution) in ASR, we propose FastCorrect, a novel NAR error correction model based on edit alignment. In training, FastCorrect aligns each source token from an ASR output sentence to the target tokens from the corresponding ground-truth sentence based on the edit distance between the source and target sentences, and extracts the number of target tokens corresponding to each source token during edition/correction, which is then used to train a length predictor and to adjust the source tokens to match the length of the target sentence for parallel generation. In inference, the token number predicted by the length predictor is used to adjust the source tokens for target sequence generation. Experiments on the public AISHELL-1 dataset and an internal industrial-scale ASR dataset show the effectiveness of FastCorrect for ASR error correction: 1) it speeds up the inference by 6-9 times and maintains the accuracy (8-14% WER reduction) compared with the autoregressive correction model; and 2) it outperforms the popular NAR models adopted in neural machine translation and text edition by a large margin.

FastCorrect: Fast Error Correction with Edit Alignment for Automatic Speech Recognition
Yichong Leng'; Xu Tan', Linchen Zhu .Jin Xu Renqian Luo', Linquan Liu
Tao Qin', Xiang- Yang Li', Edward Lin", Tie Yan Liu
University of Science and Technology of China, MMicrosoft Research Asia Microsoft Azure "Tsighuu University
[1yc123go, lrq)/maill
Smaammmalll...l www.c.cc..... edu. en, cn.iiiggaanggleueeee edw.
(xuta, taoqin, tyliu)@microooff com
[linczhu, linqul, wdwieeemeiioioidoooooeeoomo com
*j "j-uu10mmalls teinghus.ttnhttteee edu.
Abstract
Error correction techniques have been used to refine the output sentences from automatic speech recognition (ASR) models and achieve lower word error rate (WER) than original ASR outputs. Previous works usually use sequence-to- sequence model to correct an ASR output sentence autoregressively, which causes large latency and cannot be deployed in online ASR services. A straightforward solution reduce latency, inspired by non-autoregressive (NAR) neural machine translation, is to an NAR sequence generation model for ASR eπrоr correction, which, however, comes at the cost of significantly increased ASR error rate. this paper, observing distinctive error patterns and correction operations (i.c., insertion, deletion, and substitution) in ASR, we propose FastCorrect, a novel NAR error correction model based on edit alignment. In training, FastCorrect aligns each source token from an ASR output sentence to the target tokens from the corresponding ground-truth sentence based on the edit distance between the source and target sentences, and extracts the number of target tokens corresponding to each source token during edition/correetion, which is then used train length predictor and to adjust the source tokens to match the length of the target sentence for parallel generation. inference, the token number predicted by the length predictor used to adjust the source tokens for target sequence generation. Experiments on the public AISHELL-1 dataset and an internal industrial-scale ASR dataset show the effectiveness of FastCorrect for ASR eтror correction: 1) it speeds up the inference by 6-9 times and maintains the accuracy (8- 14% WER reduction) compared with the autoregressive correction model; and 2) it outperforms the popular NAR models adopted in neural machine translation and text edition by a large margin.
1 Introduction
In recent years, сттог correction techniques 5 22 26 33 have been widely adopted to refine the output sentences from an ASR model for further WER reduction. Error correction, typical sequence to sequence task, takes the sentence generated by an ASR model as the source sequence and the ground-trute -trutt sentence the target sequence, and aims correct the errors in the source sequence. Previous works on ASR error correction [22] 26 usually adopt an encoder-decoder based autoregressive generation model. While achieving good WER reduction, autoregressive models suffer from slow inference speed, and do not satisfy the latency requirements for online ASR services. For
'This work was conducted at Microsoft. Corresponding author: Tan, muta@mirrosoft.com
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

example, the latency of our internal product ASR system is about 500ms for an utterance on a single CPU, but the latency of the autoregressive correction model alone about 660ms, which is even larger than the original ASR system and unaffordable for online deployment.
Non-autoregressive (NAR) models can speed up sequence generation by generating target sequence in parallel, and attract much research attention, especially in neural machine translation (NMT) [6 g 2 4T]. Unfortunately, direct application of NAR models designed for NMT to ASR error correction leads to poor performance. According to our experiments, using popular NAR model from NMT [Р] for error correction even increases WER, i... the correction output worse than the original ASR output. Different from NMT where almost all input tokens need to be modified (i.e., translated to another language), the modifications in ASR correction are much fewer but more difficult. For example, if an ASR model achieves 10% WER, only about 10% input tokens of the correction model need to be modified, and these tokens are usually difficult to identify and correct since they have already been mistaken by the ASR model. Thus, we need to take the characteristics of ASR outputs into consideration and carefully design NAR models for ASR error correction.
In ASR error correction, the source and target tokens are aligned monotonically (unlike shuffle error in neural machine translation), and ASR accuracy usually measured by WER based on the edit distance. Edit distance provides the edit and alignment information such as insertion, deletion and substitution on the source sentence (the output of an ASR model) in order to match the target (ground- -ground-truth) sentence, which can serve as precise guidance for the NAR correction model. Based on these observations, in this paper, we propose FastCorrect, a novel NAR error correction model that leverages and benefits from edit alignment:
In training, FastCorrect first obtains the operation path (including insertion, deletion and substi- tution) through which the source sentence can be modified to target sentence by calculating the edit distance, and then extracts the token-level alignment that indicates how many target tokens correspond to each source token after the insertion, deletion and substitution operations (i.c., O means deletion, means unchanged or substitution, 22 means insertion). The token-level align- ments (token numbers corresponding to each source token) are used to train length predictor and to adjust the source tokens to match the length of the target sentence for parallel generation.
In inference, we cannot get token alignments as the ground tround-truth sentence is not available. We use the length predictor to predict the target token number for each source token and use the predicted number to adjust the source tokens, which are then fed to the decoder for target sequence generation. With this precise edit alignment, FastCorrect can correct the ASR errors more effectively, using a length predictor to locate which source token needs to be edited/corrected and how many tokens will be corrected to, and then using a decoder to correct the tokens correspondingly.
Since current ASR models have already achieved high accuracy, there might be not many errors in ASR outputs to train a correction model, even if we have large-scale datasets for ASR model training. To overcome this limitation, we use the crawled text data to construct a pscudo correction dataset by randomly deleting, inserting and substituting words in the text data. When substituting word, we use homophone dictionary considering that ASR substitution errors are mostly from homophones. Those randomly edited sentences and their original sentences compose the pseudo sentence pairs for correction model training. In this way, we first pre- traitrain FastCorrect on the large-scale pseudo correction dataset and then fine-tune the pre-trained model on the limited ASR correction dataset.
The contributions of this work are as follows:
To our knowledge, are the first to propose NAR error correction for ASR, which greatly reduces the inference latency (up to 9x) compared with its autoregressive counterpart while achieving nearly comparable accuracy. Our method also outperforms the popular NAR models adopted in machine translation and text edition by a large margin.
Inspired by the distinctive error patterns and correction operations (i.e., insertion, deletion and substitution) in ASR, we leverage edit alignments between the output text from ASR models and the ground-trutt text guide the training of NAR error correction, which is critical to FastCorrect.
The code of FastCorrect available httppp///ghhhaa..occcccoof//t//s master/FastCorren
2

---

2 Background
Error Correction In the field of natural language processing, error correction aims to correct the errors in the generated sentence by another system, such as automatic speech recognition [1
33 [I], neural machine translation [36] and optical character recognition [27]. The error correction models for ASR can be divided into two categories based whether the model can be trained in an end-to-end manner. Based on the method of statistic machine translation, Cucu et al. 4 performed -araae-basedd based machine
error correction for ASR system. D' Haro and Banchs [3] proposed to translation system to serve as a correction model for ASR. Anantaram et al. [ used a four-step method to repair the ASR model output by ontology learning. With the increasing of training corpus, end-to-end correction models are more accurate and become popular. A language model was trained for ASR correction in Tanaka et al. [39] which could exploit the long-term context and choose better results among different ASR output candidates. Mani et al. [26) utilized a Transformer-based model to train ASR correction model in autoregressive manner. Liao et al.
phrase-
22
further incorporated the
MASS (35] pre-training into ASR correction. Deliberation network [1
16 applied error correction
using both the first-pass transcripts as well as the input speech. However, all the end-to-end correction models are autoregressive and unsuitable for online deployment due to large latency, hindering the industrial application of ASR correction. Considering that there does not exit shuffle error in ASR correction, we propose a novel method to align the source sentences and target sentences towards global optimum based on edit distance, which can not only keep the matched tokens in alignment as many as possible, but also detect the substitution, deletion and insertion errors during alignment.
Non-autoregreesive Models NAR generation, which aims to speed up the inference of autoregres- sive model while achieving minimal accuracy drop, has been a popular research topic in recent years Gu et al.
88. Ma al. (24]. Shu al. 34 approached this problem with a set of latent variables. Shao et al. [32). Ghazvininejad et al. 7 Li et al. [21] developed other alternative loss functions to help the model capture target-size sequential dependencies. Wang al. (41), Stern et al.
13
23 30.
38 proposed partially parallel decoding to output multiple tokens at each decoding step. Stern et al. [37] Gu et al. 9 proposed to generate target tokens in a tree-based manner, and used dynamic insertion/delltion to iteratively refine the generated sequences based on previous predictions. FELIX [5 performed NAR text edition by aligning source sentence with target sentence greedily. However, as shown in section 5.1 directly using existing non-autoregressive models such as Gu et al. [9] and Mallinson et al. [25] cannot get satisfying results on ASR error correction. In this paper, based on the characteristics of ASR recognized text, we develop FastCorrect, which builds edit alignment between the source sentences and target sentences to guide the eтror correction.
FastCorrect
FastCorrect leverages NAR generation with edit alignment to speed up the inference of the autoregres- sive correction model. In FastCorrect, we first calculate the edit distance between the recognized text (source sentence) and the ground-truth text (target sentence). By analyzing the insertion, deletion and substitution operation in the edit distance, we can obtain the number of target tokens that correspond to each source token after edition (i.e., 0 means deletion, means unchanged or substitution, 22 means insertion). FastCorrect adopts an NAR encoder-decodd structure with a length predictor to bridge the length mismatch between the encoder (source sentence) and decoder (target sentence). The obtained number of target tokens is used to train the length predictor to predict the length of each source token after correction, and adjust each source token, where the adjusted source tokens are fed into the decoder for parallel generation. In the following subsections, we introduce the edit alignment, model structure and pre-training strategy in FastCorrect.
3.1 Edit Alignment
As shown in Figure T the edit alignment between each token in source and target sentences can be obtained through two steps: calculating the edit paths with minimum edit distance (the left and middle sub-figures), and choosing edit alignment with the highest n-gram frequency (the right sub-figure). In the next, we introduce each step in detail.
3

---

Align bt Edit Distance Matrix
Edit Path
Edit Alignment
Target
Insert
Delete
Len
A
D
Substitute
Identity
Align
Src
E
a
Src
D
A
Path a
B
3
Tgt
A
D
Len
2
D
Source
Src
Src
B
B
D
E
D
S
Path b
Tga
A
D
Ø
F
Tgt
Lon
2
0
Src
D
Allgn b2
Src
8
8
D
E
Tgi
Tgt
A
D
Ø
Delete
Substitute
Path
n-Gram
AB: 90
Align
90
0 0
Insert
Match
b:
Freq
8C:80
Freq
0
Score
6:2
Table
CD:
Scooe
62: 0
+0
Figure 1: Illustration of the edit alignment between source sentence "B B D E F" and target sentence "A C F" The left part is the edit distance matrix calculated in a recursive manner. For example, the distance 2 in row D and column B means that the edit distance between source prefix sentence "B B D" and target prefix sentence "A B" is 2. The middle part shows all the possible edit paths with the minimum edit distance. Ø stands for empty token. After filtering these paths with lower match scores, we can get all the possible token-leve level edit alignments, as shown in the right part. The alignment with the highest frequency score is selected the final edit alignment to guide the error correction.
Calculating Edit Path Edit distance measures the dissimilarity of two sentences by counting the minimum number of edit operations required transform the source sentence into the target sentence The valid edit operations include token insertion, token deletion and token substitution.
Given source sentence S (51, $2, .... SM) and target sentence T (t1, f2..... .... tN), where M and N are the lengths of source and target sentences, the edit distance between S and T can be obtained by calculating the edit distance of prefix sentences recursively. The procedure is follows:
(i min(D(i 1,5) 1, ((.
Di 1,j + 1 (s: t2))
In the above equation, 5i is the edit distance of source prefix sentence ($1, 82, .... s,) and target prefix sentence (t1, ty), 11)) true otherwise O. The boundary condition of Di is (ii, i, D(O, j.
(-) is the indicator function whose output is 1 when the condition is
The leftmost part Figure T an example of the alignment between source sentence "B D F" and target sentence "A B C D F". By recursively calculating the edit distance, we can obtain all the possible edit paths that have the minimum edit distance, In this case, we have 3 possible paths with minimum edit distance 3, as path b and e shown in the middle part of this figure.
Choosing Edit Alignment We introduce how to choose the edit alignment between each token in source sentence and target sentence based on the edit paths.
First, for the edit paths obtained from the above procedure, we calculate the match score of each path and only keep the paths with the highest match score. We define the match score of edit path as the number tokens that is not changed in this path. If an edit path has higher match score, this path is more preferred since most tokens in the source sentence can be kept. As shown in the middle part of Figure I. the match scores of the path a and b are both 3 because they both have 3 unchanged tokens: "B". "D" and "F", and the match score of path c only 2, which will be filtered
Standard edit distance usually transforms the target the source. We change the order transform the source to target to align with our scenario.
"Intuitively speaking, path does not make sense because it first changes token "D" to "C" and then changes "E" to "D". It is apparent that "D" should not be changed.

---

Second, we get the edit alignment set E (where E E represents a possible edit alignment for all tokens between the source and target sentences) from all the edit paths obtained by now by the following rules: 1) For a deletion, the source token is aligned with empty target token Ø. 2) For substitution or identity, we align the source token with the substituted token or unchanged token in the target sentence, respectively. 3) For an insertion, the target token has no source token to align with (c.g,, token "C" in path 6), and then the target token will be aligned with its corresponding left or right source token, resulting in different edit alignments (e.g., path b can generate two alignments: bl and 62, by aligning target token "C" to source token "B" (left) or "D" (right), respectively).
Third, we select the final edit alignment e from the edit aligmment set E obtained in the second step, according to the frequency of n-gram tokens of the aligned target tokens. We first build an n n-gram frequency table G' that contains the number of occurrences of every n-gram term in the training corpus, and then calculate the frequency score Freqscore(e) for each edit alignment e E on a source sentence S:
G[ len(x) Freqscore (e) = Σ Freg(e(ss)); Freq(x)= = 0. len(x) S
M
>
(2)
where e[s;] represents the target tokens aligned with the source token s, under alignment e, M is the number of tokens in source sentence, len(x) is the number of words in x and G(zz] returns the frequency of z in the n-gram table") The frequency of all 1-gram is set to 0 because we interested in differentiating token combinations. We choose the alignment e E E with the largest frequency score as the final edit alignment between each token in the source and target sentences. Doing so, we encourage the edit alignment that aligns the source token with more frequent n-gram target tokens. Taking the alignment in the rightmost part of Figure as an example, where only the first source token "B" is aligned with more than 1 target tokens (i.e., "AB"). So the frequency score of alignment is equal to the frequency of "AB" in n- g-gar table G (i.e.. 90), which is larger than that of n- n-gram token "BC" in alignment b1 and "CD" in alignment 62. As a result, the alignment is selected as the final edit alignment and the length of target tokens aligned with each source token is "2 1 0 1". respectively.
3.2 Model Structure
We use Transformer [40 as the basic model architecture of FastCorrect, as shown in Figure The encoder takes the source sentence input and outputs a hidden sequence. which is 1) fed into a length predictor to predict the number of target tokens corresponding to each source token (i.e., the edit alignment obtained in the previous subsection), and 2) used by the decoder through encoder-decoder attention. The detailed architecture of length predictor is shown in the right sub-figure of Figure 2 which is optimized with MSE loss.
Thanks to the designs of the edit alignment and length predictor in FastCorrect, the deletion and insertion errors are detected by predicting a length of 0 more than 1 the corresponding source token through the length predictor. For substitution errors, the length predicted by the length predictor is 1, which is the same as the length of unchanged/correet source token. In this condition, the substitution error can be differentiated from the unchanged token by the decoder since it is different with the target token. These designs decrease the difficulty of error correction by using the length predictor to precisely detect the error patterns and using the decoder to focus on modification.
3.3 Pre-training
The high accuracy of ASR model, whose outputs are correct in a large proportion, makes the effective training cases for correction models limited since most words in sentence correct. To overcome this problem, we construct large-scale pseudo paired data to pre-train the FastCorrect model and then fine-tune the original limited paired data. We crawl text data to construct pseudo correction dataset by randomly deleting, inserting and substituting words in text data. To simulate the ASR error as close as possible to benefit the model training, we take two considerations: 1) The word is replaced with another word with similar pronunciation from a homophone dictionary when substituting, since substitution errors in ASR usually come from homophones. 2) The probability of modifying word
In the rightmost part of Figure
E' {a, b1, 62). 62[D) CD. G[CD]=20.
5

---

Token Length
Target
Linear Layer
2
A
B
C
D
F
Dropout
Length Predictor
Attention
Linear Layer ReLU
Transformer Decoder
x5
Transformer Encoder
LN Dropout
Conv1D(k=3) ReLU
Source
Adjusted Token
Length Predictor
Figure 2: Model structure of FastCorrect.
is set to the word error rate of the ASR model. The probability distribution of deletion, insertion and substitution is set to the error distribution of the ASR model.
Experimental Setup
4.1 Datasets and ASR Models
We conduct experiments on two datasets, the public AISHELL-1 dataset and an internal product dataset.
AISHELL-1 AISHELL-1 3 is an open-source Mandarin speech corpus with 178 hours of training
data We use the ESPnet 42 toolkit to train an ASR model on AISHELL-1 dataset. Several techniques such as Conformer architecture [10), SpecAugment (29] and speed perturbation utilized to improve the performance of this ASR model, resulting in a state-of-the-an WER 4.46 and 4.83 the validation and test of AISHELL-1. The ASR model is encoder-attention- decoder model with a 12-layer conformer encoder, a 6-layer conformer decoder, hidden size 512 and conformer kernel size of 15, which is trained with cross-entropy loss decoder output and an auxiliary CTC loss on encoder output. We use NVIDIA V100 GPUs for training and the batch size 32 sentences per GPU. The output unit is Chinese character. The trained ASR models transcribe AISHELL-1 to generate the paired data for error correction.
Internal Dataset Our internal dataset is Mandarin speech corpus with 75K hours of training data, Our ASR model on internal dataset is hybrid model, where the acoustic model is a ciiency-controlee BLSTM 43 with layers and 1024 hidden units in each layer, and the language model is 5-gram model with 414 million n-grams trained with 436 billion tokens. The trained ASR models transcribe internel dataset to generate the paired data for error correction.
Pseudo Data for Pre-training We 400M crawled sentences construct the pseudo paired data for pre-training. Each word in the original sentence is noised with a probability of p. which is to the word error rate of the ASR model. For word to be noised, the probability of substitution, deletion or insertion is estimated from the transcription results of ASR model, which is mentioned in the previous paragraph.
For all the text data in the above three datasets, we learn the subword using SentencePicce [9] with a dictionary size of 40K.
4.2 Model Configurations of FastCorrect and Baseline Systems
We use the default Transformer architecture in FastCorrect, which consists of a 6-layer encoder and a 6-layer decoder with hidden size dmode! 512 and feed-forward hidden size d 1024.
"atpss//openslroor//33

---

Our length predictor consists of layers of ID convolutional network with ReLU activation and 2 linear layers to output scalar, all of which have a hidden size of 512. Each convolutional layer is followed by layer normalization [2] and dropout. The kernel size of the convolutional network 3. FastCorrect is implemented on Fairseq (28). We describe the baseline systems used in our experiments for comparison: two NAR models, LevT [ and FELIX [25]. and a Transformer based autoregressive (AR) model.
LevT We compare FastCorrect with another NAR model called Levenshtein Transformer (LevT) 2.. which also predicts the insertion and deletion of token in the decoder with multiple iterations. Different from LevT that implicitly learns the insertion and deletion to refine the target sentence, FastCorrect explicitly leverages edit distance to extract the edit alignment (insertion, deletion and substitution) between the tokens in the source and target sentences, which can be more efficient and accurate. We train LevT for error correction with the default haper-parameters in Fairsed
FELIX FELIX an NAR model for text edition 25 Different from FastCorrect that utilizes edit distance for accurate alignment, the alignment algorithm is based on finding the matched tokens between the source and target sentences greedily, which 1) will be incorrect the same token appears many times, 2) cannot perform substitution directly when editing text (i.e., need first deletion and then insertion). We implement FELIX based on the official code
AR Model For the autoregressive model, we follow the standard Transformer encoder-decoder model adopted in machine translation while keeping the parameter amount comparable with FastCor- rect. We use the standard settings and hyper-parameters for AR model training in Fairsec
4.3 Training and Inference
We train all correction models on 4 NVIDIA Tesla V100 GPUs, with a batch size of 12000 tokens. We follow the default parameters of Adam optimizer [8] and learning rate schedule in Vaswani et al.
40 All the corrections models are first pre-trained then fine-tuned on the AISHELL-1 or the internal dataset for 20 epochs. To simulate the industrial scenario, we test the inference speed of the correction models in three conditions: 1) NVIDIA P40 GPU, 2) 4-core CPU and 3) single-core CPU, where the CPU "Intel(R) XRon(R) CPU E5-2690 v4 @ 2.60GHz". The test batch size is set to 1 sentence to match the online serving environment.
the pseudo data corpus for 30 epochs and
Results
In this section, first introduce the accuracy and latency of FastCorrect, and then compare Fast- Correct with Transformer based autoregressive model, LevT and FELIX. Then we conduct ablation studies to verify the effectiveness of designs in FastCorrect, including edit alignment (length predictor) and pre-training Finally, we conduct more analyses to compare FastCorrect with other methods.
5.1 Accuracy and Latency of FastCorrect
We first report the accuracy and latency of different error correction models on AISHELL-I and the internal dataset in [ We have several observations: 1) Autoregressive (AR) correction model can reduce the WER (measured by WERR) of ASR model by 15.53% and 8.50% on the test set of AISHELL-1 and internal dataset, respectively. 2) Lev'T, a typical NAR model from NMT, achieves minor WERR on AISHELL- and even leads to WER increase on the internal dataset. Meanwhile, LevT can only speed up the inference of the AR model by 2-3 times on GPU/CPU conditions. 3) FELIX only achieves 14% WERR on AISHELL-I and 0.27% WERR on internal dataset, which is much worse compared with FastCorrect, although the inference speedup is similar. 4) Our proposed FastCorrect speeds up the inference of the AR model by 6-9 times on the two datasets on GPU/CPU conditions and achieves 8- 141 WERR, nearly comparable with the AR correction model in accuracy.
We further analyze the difference between FastCorreet, LevT and FELIX in Scction 5.4
The above
" http:://ww.ccomoooooo.oem//m///p ottprrrriiiiiiooooooooooooooeeee translation

---

Table I: The correction accuracy and inference latency of different correction models. We report the word error rate (WER), word eтror rate reduction (WERR) and latency of the autoregressive (AR) and von-autoregressivv (NAR) models (FastCorrect, LevT and FELIX). "Mlter" is hyper-parameter in Lev'T controlling max decoding iteration. The actual iteration can be smaller than "Mlter" due to early stopping.
Test Sct
Dev Set
Latency (ms/sent) on Test Set
AISHELL-1
WER
WERR
WER
WERR
GPU
CPU*4
CPU
No correction
4.83
4,46
AR model
4.08
15.53
3.80
14.80
149.5 (1x)
248.9 (1x)
531.3 (Ix)
LevT (Miter=I)
4.73
2.07
4.37
2.02
54.0 (2.8x)
82.7 (3.0x)
158.1 (3.4x)
LevT (Miter=3)
4.74
1.86
4.38
1.79
60.5 (2.5x)
83.9 (3.0x)
161.6 (3.3x)
FELIX 25
4.63
4.14
4.26
4.48
23.8 (6.3x)
41.7 (6.0x)
85.7 (6.2x)
FastCorrect
4.16
13.87
3.89
13.3
21.2 (7.1x)
40.8 (6.1x)
82.3 (6.5x)
Test Set
Dev Set
Latency (ms/sent) Test Set
Internal Dataset
WER
WERR
WER
WERR
GPU
CPU*4
CPU
No correction
11.17
11.24
AR model
10.22
8.50
10.31
8.27
191.5 (1x)
336 (1x)
657.7 (1x)
LevT (Miter1))
11.26
-0.80
11.35
-0.98
60.5 (3.2x)
102.6 (3.3x)
196.5 x3.3x)
LevT (Miter3))
11.45
-2.50
11.56
-2.85
(2.5x)
118.9 (2.8x)
248.0 (2.7x)
FELIX 25
11.14
0.27
11.21
0.27
25.9 (7.4x)
(7.8x)
90.9 (7.2x)
FastCorrect
10.27
8.06
10.35
7.92
21.5 (8.9x)
42.4 (7.9x)
88.6 (7.4x)
results demonstrate the effectiveness of FastCorreet in speeding up the inference of eπгоr correction while maintaining the correction accuracy.
5.2 Ablation Study
We conduct ablation study to verify the im- portance of the designs in FastCorrect, in- cluding the edit alignment (length predic- tor) and pre-training. For the setting with- out edit alignment, predict the length difference between the input and the target, according to which the input is adjusted softly and fed into the decoder [12]. We show the results of each setting on the test set of both datasets Table We have several observations: 1) Removing edit alignment causes large WER increase, which shows that the edit alignment is critical to guide the model correct the in the recognized text. 2) Pre-training is effective when the data
Table 2: Ablation study of each design in FastCorrect.
Internal Dataset
AISHELL-1 Dataset
train predictor
Model
No correction
11.17
4.83
AR model
10.22
4.08
Pre-training
10.26
16.01
Fine-tuning
11.70
5.28
FastCorrect
10.27
4.16
Pre-training
10.33
4.83
Fine-tuning
11.74
5.19
Edit Alignment
12.27
4.67
size in the fine-tuning stage is small. Removing pre-training causes large WER increase on the AISHELL-1 dataset, which is much smaller than the internal dataset, especially in the AR setting, 3) Fine-tuning is necessary ensure WER reduction in correction models, otherwise pre-training alone can result in worse WER than original ASR outputs. The above observations verify the effectiveness of each design in FastCorrect.
5.3 Comparison to AR Model with Shallow Decoder
deep encoder and shallow decoder can reduce the latency of the AR model while maintaining the accuracy. We compare the accuracy and latency of FastCorrect to AR models with different combinations of encoder and decoder layers the test set of both datasets in Table 3
As proposed in 14 17

---

Table 3: Comparison of FastCorrect and the AR model with deep encoder and shallow decoder. "AR 8-4" means the AR model with an 8-layer encoder and 4-layer decoder.
AISHELL-1
Internal Dataset
Model
WER
Latency (ms/sent)
WER
Latency (ms/sent)
GPU
CPU
%
GPU
CPU
No Correction
4.83
11.17
AR 6-6
4.08
149.5 (1x)
531.3 (1x)
10.26
190.6 (1x)
648.3 (1x)
AR 8-4
4.14
120.5 (1.2x)
427.6 (1.2x)
10.28
144.1 (1.3x)
542.0 (1.2x)
AR 10-2
4.23
84.0 (188)
317.6 (1.5x)
10.33
100.8 (19x)
431.2 (1.5x)
AR 11-1
4.30
66.5 (2.2x)
281.0 (1.7x)
10.44
79.1 (2.4x)
372.3 (1.7x)
FastCorrect
4.16
21.2 (7.1x)
82.3 (6.5x)
10.33
21.4 (8.9x)
86.8 (7.5x)
Table 4: Comparison of models on several metrics. We report: Pedit, among all the edited tokens, how many tokens actually need to be edited; Redit, among all the error tokens, how many tokens are edited; Pright. among
the edited tokens, how many tokens are edited to target (i.e., error conrected). Peait and Redat reflect the error-detction ability and Pright reflects the enror-corrcttoon ability. The word error rate reduction (WERR) of each model also shown.
Internal Dataset
AISHELL-1
Model
Pedit
Redie
Pright
WERR
P Pedst
Redat
Pright
WERR
AR model
78.8
17.5
50.5
8.50
83.9
33.6
54.0
15.53
LevT
57.9
16.7
33.6
-0.80
71.7
14.4
38.4
2.07
FELIX
66.3
6.5
35.3
0.27
76.7
18.3
40.6
4.14
FastCorrect
79.0
18.4
47.6
8.06
83.7
34.5
50.1
13.87
FastCorrect achieves a much larger speedup than the AR models with a deep encoder and shallow decoder while maintaining similar accuracy with the AR models. To be more specific, FastCorrect has better or comparable performance with auto-regressive model with 10 encoder layers and 2 decoder layers, while the speedup is about 5x higher.
5.4 Analysis of FastCorrect, LevT and FELIX
We perform an analysis on the ability of error detection and error correction between FastCorrect, LevT and FELIX to figure out why FastCorrect is better than LevT and FELIX.
If a source token is edited (including insertion, deletion or substitution) by a correction model, the edit result can be correct (same with the target token) incorrect (different from the target token Therefore, we can calculate several metrics to measure the advantage of a correction model: 1) Podic- among all the edited tokens, how many tokens actually need be edited. 2) Redits among all the error tokens, how many tokens are edited. 3) Pright. among all the edited tokens, how many tokens edited to target (i.e., error is corrected). Pedr and Redi measure the eroo-detection ability of correction models while Pright measures the error-correction ability. The comparison of AR model, LevT, FELIX and FastCorrect these metrics shown in Table 4
From Table we can observe that compared with FastCorrect, the error-correction ability both LevT and FELIX is inferior, since the ratio of tokens whose emrors are corrected, Pright: of FastCorrect higher than that of LevT or FELIX by large margin. Moreover, the error-detection ability of LevT the internal dataset, where the Pedit of LevT is only 57.9%.
weaker than FastCorrect, especially meaning that 42. 1 tokens edited by LevT are accurate already and these modifications only lead to WER increase. Thanks to the accurate edit alignment, FastCorrect can split the error detection and error correction into different modules, using length predictor to detect errors and decoder to
For example, for a source sentence "ADC" and target sentence "ABC", The edit result is correct if the source token "D" is edited to "B", while incorrect if edited to "E"

---

correct errors. Thus, FastCorrect can have comparable error-detection ability with AR model and high arro-ccorection ability, enabling it to nearly match the accuracy of the AR model, and greatly outperform the two baselines.
6 Conclusions
In this paper, to reduce inference latency while maintaining the accuracy of conventional autoregres- sive error correction models for ASR, we proposed FastCorrect, a non-autoregressive model that leverages edit alignments (insertion, deletion, and substitution) between the tokens in the source and target sentences to guide error correction. FastCorrect utilizes a length predictor for error detection (predicting the number of target tokens aligned to each source token to detect insertion, deletion, and substitution/identity) and a decoder for error correction (modifying the error token into the correct one). Experiments on public AISHELL-1 and large-scale internal dataset show that FastCorrect reduces inference latency an autoregressive model by 6-9 times while maintaining its accuracy, and outperforms previous NAR models proposed for machine translation and text edition, which verifies the effectiveness of FastCorrect. Although FastCorrect is proposed for ASR, its design principle that first detecting errors and then correcting errors would be beneficial for general error correction. We will explore potential extensions of FastCorrect to other tasks and enable FastCorrect to leverage multiple ASR candidates 220
References
[1] Anantaram, Amit Sangroya, Mrinal Rawat, and Aishwarya Chhabra. Repairing asr output by artificial development and ontology based learning. In IJCAI, 2018.
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. In arXiv preprint 1607v 2016.
[3] Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline. In 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech 1/O Systems and Assessment (O-COCOSDA), 2017.
[4] Horia Cucu, Andi Buzo, Laurent Besacier, and Corneliu Burileanu. Statistical error correction methods for domain-specific asr systems. In Adrian-Horia Dediu, Carlos Martín-Vine Vide, Ruslan Mitkov, and Bianca Truthe, editors, International Conference Statistical Language and Speech Processing, 2013.
[5] Luis Fernando D' o'Haro and Rafael E Banchs. Automatic correction of asr outputs by using machine translation. In INTERSPEECH, 2016.
[6] Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predies Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Laguuge Processing (EMNLL--JCNiPP), 2019.
[7] Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. Aligned cross entropy for non-autoreeressive machine translation. In International Conference on Machine Learning (ICML), 2020.
[8] Jiatao Gu, James Bradbury, Caiming Xiong. Victor O.K. Li, and Richard Socher. Non- autoregressive neural machine translation. In International Conference Learning Rep- resentations (ICLR), 2018.
[9] Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. In Advances in Neural Information Processing Systems 32 (NIPS), 2019.
[10] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-angmented transformer for speech recognition. In INTERSPEECH, 2020.
10

---

[11] Jinxi Guo, Tara N Sainath, and Ron J Weiss. A spelling correction model for end-to-end to-end speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5651-5655. IEEE, 2019.
[12] Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, and Tie- Liu. Non-autoregressive neural machine translation with enhanced decoder input. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
[13] Junliang Guo, Xu Tan, Linli Xu, Tao Qin, Enhong Chen, and Tie- Yan Liu. Fine-tuning by curriculum learning for non-autoregressive neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
[14) Tianyu He, Xu Tan, and Tao Qin. Hard but robust, easy but sensitive: How encoder and decoder perform in neural machine translation. In arXiv preprint PrXiv:1108.062255 2019.
[15] Ke Hu, Tara N Sainath, Ruoming Pang, and Rohit Prabhavalkar. Deliberation model based two-pass end-to-end speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2020.
[16] Ke Hu, Ruoming Pang, Tara N Sainath, and Trevor Strohman, Transformer based deliberation for two-pass speech recognition. In IEEE Spoken Language Technology Workshop (SLT), 2021.
[17] Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah A Smith. Deep encoder, shallow decoder: Reevaluating the speed-quality tradeoff in machine translation. In International Conference Learning Representations (ICLR), 2021.
[18] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations (ICLR), 2015.
[19] Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2018.
[20] Yichong Leng, Xu Tan, Rui Wang. Linchen Zhu, Jin Xu, Linquan Liu, Tao Qin, Xiang-Yang Li, Edward Lin, and Tie- Yan Liu. Fastcorrect 2: Fast error correction on multiple candidates for automatic speech recognition. In Findings of EMNLP, 2021.
[21] Zhuohan Li, Zi Lin, Di He, Fei Tian, QIN Tao, WANG Liwei, and Tie- Yan Liu. Hint- based training for non-autoregressive machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019.
[22] Junwei Liao, Sefik Emre Eskimez, Liyang Lu, Yu Shi, Ming Gong, Linjun Shou, Hong Qu, and Michael Zeng. Improving readability for automatic speech recognition transcription. In arXiv preprint ar 2004:40000.44438, 2020.
[23] Jinglin Liu, Yi Ren, Xu Tan, Chen Zhang, Tao Qin, Zhou Zhao and Tie-Ya Yan Liu. Task-level curriculum learning for non-autoregressive neural machine translation. In ISCAI, 2020.
[24] Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. Flowseq: Non- autoregressive conditional sequence generation with generative flow. In Proceedings the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing DEMNLL--JCNLLS 2019.
[25] Jonathan Mallinson, Aliaksei Severyn, Eric Malmi, and Guillermo Garrido. FELIX: Flexible text editing through tagging and insertion. In Findings of the Association for Computational Linguistics: EMNLP 2020, 2020.
[26] Anirudh Mani, Shruti Palaskar, Nimshi Venkat Meripo, Sandeep Konam, and Florian Metze. Ast error correction and domain adaptation using machine translation. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), 2020.
[27] Kareem Mokhtar, Syed Saqib Bukhari, and Andreas Dengel. Ocr error correction: State-of-the- art vs an nmt-based approach. In 13th IAPR International Workshop on Document Analysis Systems (DAS), 2018
11

---

[28] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), 2019.
[29] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. In INTERSPEECH, 2019.
[30] Yi Ren, Jinglin Liu, Xu Tan, Zhou Zhao, Sheng Zhao, and Tie- Yan Liu. A study of non- autoregressive model for sequence generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
[31] Eric Ringger and James Allen. Error correction via post-pro prst-prcessor for continuous speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1996.
[32] Chenze Shao, Jinchao Zhang. Yang Feng, Fandong Meng, and Jie Zhou. Minimizing the bag-of-ngrams difference for non-autoregressive neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
[33] Prashanth Gurunath Shivakumar, Haoqi Li, Kevin Knight, and Panayiotis Georgiou. Learning from past mistakes: improving automatic speech recognition output via noisy-clean phrase context modeling. In APSIPA Transactions Signal and Information Processing, 2019.
[34] Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. Latent-variable non- autoregressive neural machine translation with deterministic inference using a delta posterior. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
[35] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Liu. Mass: Masked sequence to sequence pre-training for language generation. In International Conference on Machine Learning (ICMI). 2019.
[36] Kaitao Song, Xu Tan, and Jianfeng Lu. Neural machine translation with error correction. In IJCAI, 2020.
[37] Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. Insertion transformer: Flexible sequence generation via insertion operations. In International Conference on Machine Learning (ICML), 2019.
[38] Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models. In Advances in Neural Information Processing Systems 32 (NIPS), 2019.
[39] Tomohiro Tanaka, Ryo Masumura, Hirokazu Masataki, and Yushi Aono. Neural error corrective language models for automatic speech recognition. In INTERSPEECH, 2018
[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems 30 (NIPS), 2017.
[41] Chunqi Wang, Ji Zhang, and Haiqing Chen. Semiiautoregressive neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018
[42] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduch- intala, and Tsubasa Ochiai. Espnet: End-to-end speech processing toolkit. In INTERSPEECH, 2018.
[43] Yu Zhang, Guoguo Chen, Dong Yu, Kaisheng Yaco, Sanjeev Khudanpur, and James Glass. Highway long short-term memory mns for distant speech recognition. In IEEE International Conference on Acoustics. Speech and Signal Processing (ICASSP), 2016.
12


Reviews:
[{"review_id": "nEkJBYGbyvv", "reviewer": "Reviewer_gLsB", "summary": "This paper presents FastCorrect, a non-autoregressive text-editing model for ASR Error Correction. The authors propose a new text-editing architecture that is based on (1) sequence tagging model that predicts the final length of every source token in the target sentence and (2) a non-autoregressive decoder that predicts the target tokens in-parallel. To generate the training data, the source and the target sentences are aligned by calculating the edit distance and computing the edit path from the source to the target. Heuristics are applied to choose the path among all the paths with the minimal edit distance. The chosen path is used to create the labels for the length predictor and the non-autoregressive decoder.\n\nThe authors also perform data augmentation, generating data with synthetic errors using a homophone dictionary. This data is used to pre-train FastCorrect. Unfortunately it seems that this part is not reproducible, the crawled data is not published and it is not clear what homophone dictionary was used (I can only assume it is not open source).\n\nThe authors test FastCorrect on a public dataset and another internal dataset. On the public dataset there is a reduction of 0.67 WER. The authors then compare the WER and inference speed between FastCorrect and 3 other models: 2 publicly available non-autoregressive text-editing models (LevT and FELIX) and an autoregressive Transformer-based encoder-decoder model. FastCorrect shows superior performance in terms of WER compared to the other 2 non-autoregressive models and ~9 times faster inference than the autoregressive model.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written. ASR Error Correction is still an open area of research and new contributions are always welcome. The modeling idea is elegant and intuitive. The alignment algorithm is well presented and clearly explained (Figure 1 and the relevant description in section 3.1). It is encouraging to see that a text post-processing model can decrease WER. \n\nThe significance of the WER improvement is not clear to me. I am not sure whether 0.67 counts as a good improvement in this particular setup and whether the differences between the different error correction models are statistically significant. The evaluation is done on errors from a single ASR model, the results could be more convincing if they were evaluated using several (at least 2) ASR models. Given the fact that the  Error Correction model uses the same data as used to train the ASR model (​​AISHELL-1 training set), it is surprising to see that the ASR model could be corrected with a simple text-based model, even though the ASR model has acoustic information that is not available to the text-editing model. It could be really interesting to see a data analysis, or at least a case study, to understand what types of errors are actually being corrected.\n\nWhile I understand that using an AR model with the same number of parameters is great to back up the claims about inference times, it could be interesting to see a WER of a state-of-the-art pre-trained sequence-to-sequence model (T5? BART?). If I would want to optimize WER I would start with a state-of-the-art pre-trained model, then if I would consider a NAR model for faster inference I would want to know the WER gap between them.\n\nI did not read about LevT but from reading the FELIX paper, I think that claim #2 about FELIX is inaccurate, in Figure 2 in the paper (https://aclanthology.org/2020.findings-emnlp.111.pdf) it seems that there is a DEL-INS tag that allows substitutions. I also did not understand claim #1 and hoped the authors can provide a clarification. Why is the alignment incorrect if the same token appears many times? Maybe an example?\n\nI find the analysis in section 5.4 problematic. It evaluates the error detection and correction on the sentence-level. Sentences can contain many errors, detecting or correcting part of them should count. For example if the sentence contained 10 errors, detecting/fixing 9 or 1 is not the same.\n\nSection 3.1 lists different heuristics for choosing between the paths with the minimum edits (choosing the version that keeps most of the source tokens, using n-gram frequency table). While there is a clear and well explained intuition behind those heuristics, they complicate the algorithm and it is not clear whether they are actually necessary. An ablation study with a random shortest path could be helpful.\n\nReproducibility:\n\nFrom the appendix it seems that the authors released only the recognition results of the ASR model and their evaluation scripts. It seems (from the description in the appendix, I did not look at the code) that the full training code (ASR and FastCorrect) was not released, so is the pre-trained or fine tuned version of FastCorrect. The authors claim that they will release those once the paper is accepted.\nMy main concern remains about the data used to pre-train FastCorrect. Even if the authors will release the data that they committed to release, the pre-training data is not published and it is not clear what homophone dictionary was used (probably not open source). This makes it impossible for future researchers to reproduce the results and build on this work.\nThis is crucial because in Table 2 we can see that without the pre-training there is actually no WER improvement on the public dataset using FastCorrect.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A agree with the authors that there is no obvious potential negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "None that I could think of.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "l6F8oXIsGa", "reviewer": "Reviewer_CpNc", "summary": "This paper addresses the topic of designing a non-autoregressive approach for correcting ASRs’ predictions. The motivation of applying a non-autoregressive approach is to allow parallel error corrections for the given hypothesis and reduce the inference latency. The proposed model has a length predictor which infer the target token length per input token, and use that info to map input tokens to target tokens and construct the final hypothesis. During training this target token length is calculated based on finding the minimal edit distance between source and target. In most cases there are multiple editing path with the same editing distance, and the work proposes to pick the one with highest alignment frequency between source and target. The proposed non-autoregressive approach significantly reduces the latency compared to autoregressive models, and provides lower WERs compared to existing non-autoregressive approaches.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed approach infer per-token target length based on edit distance which is a nice design, and the empirical results showed significant improvement compared with other correction-based approaches. On the other hand, alternative to correcting the hypothesis with a non-autoregressive model, one can use the non-autoregressive model to rerank the hypothesis. The reranking approach does not need to infer the target length, and the training and inference pipelines are simpler. The paper lacks the comparisons with reranking approaches, and it’s unclear whether the proposed approach is more effective than reranking.\n\nThe framework is a minor extension of previous work, making its novelty limited.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper needs to compare with more related approaches to inform whether the proposed work provides benefits over existing approaches.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "7R2fcrR5p6i", "reviewer": "Reviewer_GvdZ", "summary": "In this work the authors consider the problem of correcting the transcripts produced by the ASR system in the first pass. In previous works, this has been explored with attention-based encoder-decoder models which must process hypotheses sequentially left-to-right by feeding each output token. However, instead the authors propose to investigate non-autoregressive models where all outputs can be processed in parallel. This is achieved by first predicting the number of output characters corresponding to each source character and then simply modifying the input source sequence by repeating/deleting tokens as appropriate. The main benefit of the proposed approach is that it allows for significant savings in latency during inference.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is generally well written and the main ideas are easy to follow. The central theme of the paper is relatively simple; I particularly liked the ideas around mapping the transcript to a single alignment sequence in a deterministic way to remove the inherent ambiguity in the definition of edit-alignments corresponding to the minimum edit distance. That being said, I have a few suggestions which I think the authors could address to strengthen the paper:\n\n- **Related work**: I would also like to suggest that the authors include references to works which apply error correction using both the first-pass transcripts as well as the input speech in the so-called deliberation framework [Hu et al., 20] [Hu et al., 21]. I would also suggest that the authors reference this work [Guo et al., 19] which applies an error correction model in ASR.\nK. Hu, R. Pang, T. N. Sainath and T. Strohman, \"Transformer Based Deliberation for Two-Pass Speech Recognition,\" 2021 IEEE Spoken Language Technology Workshop (SLT), 2021.\nK. Hu, T. N. Sainath, R. Pang and R. Prabhavalkar, \"Deliberation Model Based Two-Pass End-To-End Speech Recognition,\" ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020.\nJ. Guo, T. N. Sainath and R. J. Weiss, \"A Spelling Correction Model for End-to-end Speech Recognition,\" ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 5651-5655, doi: 10.1109/ICASSP.2019.8683745.\n\n- **Length prediction modeling and accuracy**: Something that the paper does not mention in sufficient detail is how the length prediction model is defined. Do you have a softmax over N possible values corresponding to 0 .. N-1? What size of N do you use? Please mention in the text. Also, what is the accuracy of the length prediction model on a held-out set? \n\n- **Defining Latency**: The authors mention that one of the drawbacks of the AR-based model for spelling correction is that it has a very high latency and that it cannot be used for “online ASR tasks”. I think it is important to define these terms in the paper when they are introduced. For example, many works refer to the per-token latencies (which will still be extremely high in your setup since the error correction applies to the whole hypothesis). Also, if the encoders/decoders used LSTMs instead of transformers, then the sequence could in principle be encoded while the hypothesis was being decoded which would save some latency. I would suggest that the authors discuss more details in the paper.\n\n- **Experimental Setup**: A number of details are missing in the experimental setup in Section 4. For example, what is the structure of the base ASR model? Is it a hybrid system or one based on encoder-decoder architectures? How was it trained? What output units does the model use, etc. Please clarify these details in the paper. Are the AI-shell models decoded without an LM?\n\n- **Section 5.2**: Minor comment: The section which describes the ablation study is extremely brief, which makes it somewhat hard to fully understand. For example, the setup without edit alignment could be described in more detail, if possible. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't see any negative societal impacts of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overview: The paper proposes a model that corrects ASR errors in a post-processing step. As reviewer GvdZ points out, there has been considerable recent work in this area. The model transforms the ASR output using a series of edit operations carried out using an encoder-decoder model which is augmented with duration (per input token). Empirical evaluations demonstrate show a small improvement over other comparable approaches.

Reviews: The author responses have addressed most of the concerns raised by two of the reviewers. The third reviewer felt the novelty of the proposed approach was incremental and limited. The meta review has taken into considerations the points raised by the three reviewers and the author responses. 
Decision: Accept (Poster)