Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new non-autoregressive model that incorporates the pronunciation information with text information for the task of ASR error correction. The proposed model uses a transformer architecture for two encoders and a decoder, it adopts three different fusion approaches to combine the representations from phoneme encoder and text encoder for the target length prediction. The decoder computes the cross attention with both encoders sequentially and predicts the target words. 
strength_and_weaknesses: Strength:

1.The idea of phoneme fusion and cross attention strategy is simple yet effective. The model outperforms FastCorrect on English datasets, and achieves SOTA results on English corpus.

2. The paper compares three feature fusion approaches, and shows that cross attention is effective for text and phoneme features.

3. Experimental results are based on comparisons on 3 English ASR systems to show the robustness of the proposed approaches.

Weakness：
1. A Phoneme MHA module is added behind the original Text MHA module in the Transformer decoder block, but the ablation study shows that without this phoneme attention, the model performance roughly remains the same as adding it. So with more calculation on the phoneme attention, the model barely gets any improvement from it.
2. The improvements on some datasets are minor. All of the fusion approaches of combining the phoneme embedding and text embedding don’t consider the relationship between each word and its corresponding pronunciation, which may be important for the words error correction.
3. The novelty of proposed framework is not significant. The architecture does not seem that there is too much difference with FastCorrect except the phoneme embedding, and two extra cross attention. Besides that, model architecture and train process are almost the same.
4. Phoneme feature is commonly used in error correction [1, 2], and from the ablation experiments we see a lot of contributions are from phoneme feature. To my knowledge cross-attention feature fusion can be seen in Image field previously [3].

[1] Wang et al., ASR Error Correction with Augmented Transformer for Entity Retrieval. Interspeech 2020.

[2] Fang et al., Non-Autoregressive Chinese ASR Error Correction with Phonological Training. NAACL 2022.

[3] Bai et al., TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers. CVPR 2022.


clarity,_quality,_novelty_and_reproducibility: The paper is well written and the proposed multi-modal fusion method is easy to follow. But compared with FastCorrect, the novelty is quite limited. 

summary_of_the_review: This paper proposes a network of NAR Transformer with a phoneme encoder. Compared with FastCorrect, the improvement is limited, but it requires more latency than FastCorrect. Besides, the idea of incorporating phoneme information to gain better ability of error correction is useful but not very creative, since many text correction approaches have been proposed in recent years with similar ideas.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The task is automatic speech recognition (ASR). The paper proposed PATCorrect, a new correction model which operates on the output of another speech recognition model and tries to improve the output by correcting errors.

The proposed model works in a non-autoregressive way and thus can run in parallel, efficiently on hardware like GPUs.

It uses three different pretrained models to test the correction model on. The best model is a Conformer, which gets 10.03% WER on CommonVoice Test without correction, and the best PATCorrect correction gets down to 9.78% WER. The other models are very weak in comparison, but then the relative improvements are larger.
strength_and_weaknesses: Strengths:

- An interesting novel model for error correction.
- I think the use of phoneme inputs is novel.

Weaknesses:

- No comparison to a standard language model fusion? See below.
- The code is not published?
- The analysis and ablations study is too short.
- Ablation studies only with weak model.
- Only tested on speech recognition.
- Many aspects are unclear. See below.
- Studies are on Common Voice, but LibriSpeech would be better, as this is what most people know much better.

clarity,_quality,_novelty_and_reproducibility: Where does the phoneme sequence come from? It seems just from the recognized word sequence. So no pronunciation variants? Also, why this way, why not use the ASR encoder output or so? Also, how relevant are the phonemes anyway? There is a bit of ablation study on this, but this comes very short, and was only done for some of the very weak models. It should be done for Conformer.

Related work: It only addresses error correction. But just shallow fusion / log-linear combination with a standard language model (LM), how is that really different? As far as I know, not much really can be gained from such error correction models over standard LM fusion with a good LM.

In general, this is really an important comparison, to make use of an external LM, in comparison to such error correction, or in combination, or other variations. Further, it makes sense to test different model architectures, such as CTC, Transducer or Attention-based Encoder-Decoder.

It seems the error correction model is just trained on transcriptions, not more. So this is much weaker than a LM trained on text-only data, which is usually available in much larger quantities. But the error correction model could maybe also trained on text-only data? This should be addressed.

It's even not clear, those pretrained models, are those Transducer, CTC, att-based enc-dec?
How is decoding done? Unclear.

Pretrained models, but not explained how pretrained or which pretrained models exactly? Are they public?

Speed comparison: I don't really know: Does this only measure PATCorrect itself, or the whole decoding? What is actually the underlying model and decoding procedure?

Latex wrong math usage, use \operatorname for all whole-word-functions like softmax, Addition, Max, etc.
Latex wrong math usage, use \textrm or so for text subscripts like PATCorrect etc.

summary_of_the_review: There are too many weaknesses, as explained above. This needs more work.

In general, this paper seems more appropriate for a speech conference (Interspeech etc)? While in principle it might be applicable to other tasks, I think this should be tested, and I think this is necessary for publication on ICLR. The novelty and scope is otherwise clearly not high enough. Otherwise, even for speech conferences, I think the quality is not high enough yet, and this needs to address the weaknesses I explained before.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: PATCorrect is a non-autoregressive (NAR) error correction module.  PATCorrect is distinguished by using aligned phoneme and word inputs for making correction decisions.  Evaluation across a number of input ASR models demonstrates that PATCorrect outperforms FastCorrect, a similar NAR correction model which does not use phone information.
strength_and_weaknesses: Strength
* The use of phonetic information in error correction is well motivated.
* The architecture is well described and well motivated.

Weakness
* The improvements offered by PATCorrect are somewhat modest.  There is a small, but consistent improvement over FastCorrect.  But the performance is still substantially worse than AR Transformer. 
* The reporting of performance is, at some points, strained.  1) The decision to use F_0.25 instead of a more common detection metric of F_1 is surprising, but the authors gesture toward the bias toward Precision.  However, this decision is less convincing when observing that FactCorrect has a better F_1 score than PATCorrect, though PATCorrect has a higher F_0.25 score.   2) The reported improvement of 20% in the abstract and introduction is a relative improvement to the relative improvement to WER (the primary metric for ASR).  This is not clear in the abstract.  The absolute WER difference between FastCorrect and PATCorrect is between 1 and 2%. WERR (relative improvement to WER) may be a more relevant measure for error correction, however, since this is already a relative measure, absolute difference to WERR are a more interpretable description of performance differences.  The underlying work is strong, but these presentation decisions leave the impression that the work is presented in an overly optimistic manner.
clarity,_quality,_novelty_and_reproducibility: * Mostly quite clear.  One clarification question: Where does the phone sequence used for correction come from?  Is it generated from the ASR system directly, or is it a post-processing of the ASR hypothesis?  Also, Is there any benefit from consuming an N-best list of hypotheses from the ASR models?  While not all ASR models generate phone sequences, some do (or can be trained to simultaneously with words).  All decoders can produce N-best lists.

Since the models explored are full-context models (i.e. not streaming), it's not clear that latency is critical to their performance.  Low latency error correction would be more compelling on a streaming model.

Quality: Strong
Novelty: Sufficient Novelty
Reproducibility: Could reproduce.
summary_of_the_review: The central contribution of the work is strong, and clearly described.  The evaluations themselves are good.  However, the presentation could be more tempered to more accurately demonstrate the strengths and weaknesses of the approach.  A stronger motivation for the use of NAR error correction will also provide additional context for the contribution of the work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposed to integrate phoneme information into the ASR error correction by fusing the phoneme information with the text information, under the non-autoregressive (NAR) framework. 

Strength: The method is very well motivated, and the authors showed that the proposed method works well on 3 different types of pre-trained models, better than FastCorrect. 

 Weakness:
The novelty is not significant. In the first round rebuttal, the authors argued they are the first to integrate phoneme information into the NAR error correction. However, after the reviewer pointed out a prior work also did so, then the authors switched their tone by claiming they applied the phoneme assisted error correction to English while the prior work was on Mandarin. To me, such application to a new language is not the novelty justifying the publication as ICLR. 

While the authors answered most weakness challenges in their rebuttal, there are some answers not satisfying. For example, integrating acoustic information from encoder instead of using text/phoneme information should further improve correction accuracy. However, the authors didn't answer this challenge well. 
Regarding the increased latency of the proposed method, the authors argued they are use experimental code, while in the future they should be able to optimize the latency. The answer is not satisfactory because we should judge the paper based on the current status instead of the promise of future. 
The authors also stated in the rebuttal of not working on Librispeech is because the task has very low WER. This worries me a little because our hope is to further the best ASR model regardless the baseline WER. It seems the authors are picking some scenarios (e.g., training/testing mismatch) that have large room of improvement, and show the effectiveness of the proposed method on those scenarios. 

The experiment baseline setup also has issues. The authors should have external LM fusion as a strong baseline to compare. But the authors just simply argued that external LM fusion is an AR method. However, the readers have more interests in the best error correction performance instead of whether the method is AR or NAR. 
Furthermore, the authors used FastCorrect as the baseline. Why not the improved FastCorrect 2? 
Leng, Yichong, et al. "Fastcorrect 2: Fast error correction on multiple candidates for automatic speech recognition." arXiv preprint arXiv:2109.14420 (2021).



  
justification_for_why_not_higher_score: There are lots of weakness in this paper as listed in the meta review. The authors didn't fully address the reviewers' concerns. The experiment baseline is not strong enough.
Decision: Reject