Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this work the authors consider the problem of correcting the transcripts produced by the ASR system in the first pass. In previous works, this has been explored with attention-based encoder-decoder models which must process hypotheses sequentially left-to-right by feeding each output token. However, instead the authors propose to investigate non-autoregressive models where all outputs can be processed in parallel. This is achieved by first predicting the number of output characters corresponding to each source character and then simply modifying the input source sequence by repeating/deleting tokens as appropriate. The main benefit of the proposed approach is that it allows for significant savings in latency during inference.

main_review: Overall, the paper is generally well written and the main ideas are easy to follow. The central theme of the paper is relatively simple; I particularly liked the ideas around mapping the transcript to a single alignment sequence in a deterministic way to remove the inherent ambiguity in the definition of edit-alignments corresponding to the minimum edit distance. That being said, I have a few suggestions which I think the authors could address to strengthen the paper:

- **Related work**: I would also like to suggest that the authors include references to works which apply error correction using both the first-pass transcripts as well as the input speech in the so-called deliberation framework [Hu et al., 20] [Hu et al., 21]. I would also suggest that the authors reference this work [Guo et al., 19] which applies an error correction model in ASR.
K. Hu, R. Pang, T. N. Sainath and T. Strohman, "Transformer Based Deliberation for Two-Pass Speech Recognition," 2021 IEEE Spoken Language Technology Workshop (SLT), 2021.
K. Hu, T. N. Sainath, R. Pang and R. Prabhavalkar, "Deliberation Model Based Two-Pass End-To-End Speech Recognition," ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020.
J. Guo, T. N. Sainath and R. J. Weiss, "A Spelling Correction Model for End-to-end Speech Recognition," ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 5651-5655, doi: 10.1109/ICASSP.2019.8683745.

- **Length prediction modeling and accuracy**: Something that the paper does not mention in sufficient detail is how the length prediction model is defined. Do you have a softmax over N possible values corresponding to 0 .. N-1? What size of N do you use? Please mention in the text. Also, what is the accuracy of the length prediction model on a held-out set? 

- **Defining Latency**: The authors mention that one of the drawbacks of the AR-based model for spelling correction is that it has a very high latency and that it cannot be used for “online ASR tasks”. I think it is important to define these terms in the paper when they are introduced. For example, many works refer to the per-token latencies (which will still be extremely high in your setup since the error correction applies to the whole hypothesis). Also, if the encoders/decoders used LSTMs instead of transformers, then the sequence could in principle be encoded while the hypothesis was being decoded which would save some latency. I would suggest that the authors discuss more details in the paper.

- **Experimental Setup**: A number of details are missing in the experimental setup in Section 4. For example, what is the structure of the base ASR model? Is it a hybrid system or one based on encoder-decoder architectures? How was it trained? What output units does the model use, etc. Please clarify these details in the paper. Are the AI-shell models decoded without an LM?

- **Section 5.2**: Minor comment: The section which describes the ablation study is extremely brief, which makes it somewhat hard to fully understand. For example, the setup without edit alignment could be described in more detail, if possible. 

limitations_and_societal_impact: I don't see any negative societal impacts of this work.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper addresses the topic of designing a non-autoregressive approach for correcting ASRs’ predictions. The motivation of applying a non-autoregressive approach is to allow parallel error corrections for the given hypothesis and reduce the inference latency. The proposed model has a length predictor which infer the target token length per input token, and use that info to map input tokens to target tokens and construct the final hypothesis. During training this target token length is calculated based on finding the minimal edit distance between source and target. In most cases there are multiple editing path with the same editing distance, and the work proposes to pick the one with highest alignment frequency between source and target. The proposed non-autoregressive approach significantly reduces the latency compared to autoregressive models, and provides lower WERs compared to existing non-autoregressive approaches.
main_review: The proposed approach infer per-token target length based on edit distance which is a nice design, and the empirical results showed significant improvement compared with other correction-based approaches. On the other hand, alternative to correcting the hypothesis with a non-autoregressive model, one can use the non-autoregressive model to rerank the hypothesis. The reranking approach does not need to infer the target length, and the training and inference pipelines are simpler. The paper lacks the comparisons with reranking approaches, and it’s unclear whether the proposed approach is more effective than reranking.

The framework is a minor extension of previous work, making its novelty limited.
limitations_and_societal_impact: The paper needs to compare with more related approaches to inform whether the proposed work provides benefits over existing approaches.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents FastCorrect, a non-autoregressive text-editing model for ASR Error Correction. The authors propose a new text-editing architecture that is based on (1) sequence tagging model that predicts the final length of every source token in the target sentence and (2) a non-autoregressive decoder that predicts the target tokens in-parallel. To generate the training data, the source and the target sentences are aligned by calculating the edit distance and computing the edit path from the source to the target. Heuristics are applied to choose the path among all the paths with the minimal edit distance. The chosen path is used to create the labels for the length predictor and the non-autoregressive decoder.

The authors also perform data augmentation, generating data with synthetic errors using a homophone dictionary. This data is used to pre-train FastCorrect. Unfortunately it seems that this part is not reproducible, the crawled data is not published and it is not clear what homophone dictionary was used (I can only assume it is not open source).

The authors test FastCorrect on a public dataset and another internal dataset. On the public dataset there is a reduction of 0.67 WER. The authors then compare the WER and inference speed between FastCorrect and 3 other models: 2 publicly available non-autoregressive text-editing models (LevT and FELIX) and an autoregressive Transformer-based encoder-decoder model. FastCorrect shows superior performance in terms of WER compared to the other 2 non-autoregressive models and ~9 times faster inference than the autoregressive model.

main_review: The paper is well written. ASR Error Correction is still an open area of research and new contributions are always welcome. The modeling idea is elegant and intuitive. The alignment algorithm is well presented and clearly explained (Figure 1 and the relevant description in section 3.1). It is encouraging to see that a text post-processing model can decrease WER. 

The significance of the WER improvement is not clear to me. I am not sure whether 0.67 counts as a good improvement in this particular setup and whether the differences between the different error correction models are statistically significant. The evaluation is done on errors from a single ASR model, the results could be more convincing if they were evaluated using several (at least 2) ASR models. Given the fact that the  Error Correction model uses the same data as used to train the ASR model (​​AISHELL-1 training set), it is surprising to see that the ASR model could be corrected with a simple text-based model, even though the ASR model has acoustic information that is not available to the text-editing model. It could be really interesting to see a data analysis, or at least a case study, to understand what types of errors are actually being corrected.

While I understand that using an AR model with the same number of parameters is great to back up the claims about inference times, it could be interesting to see a WER of a state-of-the-art pre-trained sequence-to-sequence model (T5? BART?). If I would want to optimize WER I would start with a state-of-the-art pre-trained model, then if I would consider a NAR model for faster inference I would want to know the WER gap between them.

I did not read about LevT but from reading the FELIX paper, I think that claim #2 about FELIX is inaccurate, in Figure 2 in the paper (https://aclanthology.org/2020.findings-emnlp.111.pdf) it seems that there is a DEL-INS tag that allows substitutions. I also did not understand claim #1 and hoped the authors can provide a clarification. Why is the alignment incorrect if the same token appears many times? Maybe an example?

I find the analysis in section 5.4 problematic. It evaluates the error detection and correction on the sentence-level. Sentences can contain many errors, detecting or correcting part of them should count. For example if the sentence contained 10 errors, detecting/fixing 9 or 1 is not the same.

Section 3.1 lists different heuristics for choosing between the paths with the minimum edits (choosing the version that keeps most of the source tokens, using n-gram frequency table). While there is a clear and well explained intuition behind those heuristics, they complicate the algorithm and it is not clear whether they are actually necessary. An ablation study with a random shortest path could be helpful.

Reproducibility:

From the appendix it seems that the authors released only the recognition results of the ASR model and their evaluation scripts. It seems (from the description in the appendix, I did not look at the code) that the full training code (ASR and FastCorrect) was not released, so is the pre-trained or fine tuned version of FastCorrect. The authors claim that they will release those once the paper is accepted.
My main concern remains about the data used to pre-train FastCorrect. Even if the authors will release the data that they committed to release, the pre-training data is not published and it is not clear what homophone dictionary was used (probably not open source). This makes it impossible for future researchers to reproduce the results and build on this work.
This is crucial because in Table 2 we can see that without the pre-training there is actually no WER improvement on the public dataset using FastCorrect.
limitations_and_societal_impact: A agree with the authors that there is no obvious potential negative societal impact.
review_title: null

Meta-review: Overview: The paper proposes a model that corrects ASR errors in a post-processing step. As reviewer GvdZ points out, there has been considerable recent work in this area. The model transforms the ASR output using a series of edit operations carried out using an encoder-decoder model which is augmented with duration (per input token). Empirical evaluations demonstrate show a small improvement over other comparable approaches.

Reviews: The author responses have addressed most of the concerns raised by two of the reviewers. The third reviewer felt the novelty of the proposed approach was incremental and limited. The meta review has taken into considerations the points raised by the three reviewers and the author responses. 
Decision: Accept (Poster)