PATCorrect: Non-autoregressive Phoneme-augmented Transformer for ASR Error Correction
Authors: Ziji Zhang
Zhehui Wang
Rajesh Kamma
Sharanya Eswaran
Narayanan Sadagopan
Venue: Submitted to ICLR 2023 (2023)

Abstract: Speech-to-text errors made by automatic speech recognition (ASR) system negatively impact downstream models relying on ASR transcriptions. Language error correction models as a post-processing text editing approach have been recently developed for refining the source sentences. However, efficient models for correcting errors in ASR transcriptions that meet the low latency requirements of industrial grade production systems have not been well studied. In this work, we propose a novel non-autoregressive (NAR) error correction approach to improve the transcription quality by reducing word error rate (WER) and achieve robust performance across different upstream ASR systems. Our approach augments the text encoding of the Transformer model with a phoneme encoder that embeds pronunciation information. The representations from phoneme encoder and text encoder are combined via multi-modal fusion before feeding into the length tagging predictor for predicting target sequence lengths. The joint encoders also provide inputs to the attention mechanism in the NAR decoder. We experiment on 3 open-source ASR systems with varying speech-to-text transcription quality and their erroneous transcriptions on 2 public English corpus datasets. Results show that our PATCorrect (Phoneme Augmented Transformer for ASR error Correction) consistently outperforms state-of-the-art NAR error correction method on English corpus across different upstream ASR systems. For example, PATCorrect improves WER reduction by 20% compared to methods using text only modality and also achieves an inference latency comparable to other NAR models at tens of millisecond scale, especially on GPU hardware, while still being 4.2 - 6.7x times faster than autoregressive models on Common Voice and LibriSpeech datasets. 

Under review as a conference paper at ICLR 2023
PATCORRECT: NON-AUTOREGRESSIVE PHONEME AUGMENTED TRANSFORMER FOR ASR ERROR COR- RECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Speech-to-text errors made by automatic speech recognition (ASR) system nega- tively impact downstream models relying on ASR transcriptions. Language error correction models as a post-processing text editing approach have been recently developed for refining the source sentences. However, efficient models for cor- recting errors in ASR transcriptions that meet the low latency requirements industrial grade production systems have not been well studied. In this work, we propose a novel non-autoregressive (NAR) error correction approach to improve the transcription quality by reducing word error rate (WER) and achieve robust performance across different upstream ASR systems. Our approach augments the text encoding of the Transformer model with phoneme encoder that em- beds pronunciation information. The representations from phoneme encoder and text encoder are combined via multi-modal fusion before feeding into the length tagging predictor for predicting target sequence lengths. The joint encoders also provide inputs to the attention mechanism the NAR decoder. We experiment on 3 open-source ASR systems with varying speech-to-text transcription quality and their erroneous transcriptions on public English corpus datasets. Results show that PATCorrect (Phoneme Augmented Transformer for ASR error Cor- rection) consistently outperforms state-of- ofate--f-thh-aae the-art NAR error correction method on English corpus across different upstream ASR systems. For example, PATCorrect achieves 11 11.62% WER reduction (WERR) averaged on ASR systems compared 9.46 % WERR achieved by other method using text only modality and also achieves an inference latency comparable to other NAR models at tens of mil- lisecond scale, especially on GPU hardware, while still being 4.2 6.7x times faster than autoregressive models on Common Voice and LibriSpeech datasets.
INTRODUCTION
Automatic speech recognition (ASR) models transcribe human speech into readable text. It has many applications including real-time captions and meeting transcriptions. ASR model is also critical component in large-scale natural language processing (NLP) systems like Amazon Alexa, Google Home and Apple Siri. Transcribed text serves as input for downstream models such as intent detection voice assistants and response generation voice chatbots. Errors made in speech-to- text ASR transcriptions can severely impact the accuracy of downstream models and thus lower the performance of the entire NLP system.
Recent advances in ASR systems using Transformer Gulati et al. (2020): Tüske et al. (2021) and CNN based models Li et al. (2019) have achieved state-of-the-arr (SOTA) accuracy as measured by word error rate (WER). However, due the complexity of human natural language and the quality of speech audios, even SOTA ASR systems can still make unavoidable and unrecoverable errors such phonetic confusion between transcriptions, error correction models are applied to the outputs from ASR systems to detect and correct errors.
expressions. To improve the quality of ASR
ASR error correction can be formulated as a transcribed text input source sequence and the ground-truth speech-to-text transcription target
generation task, taking the ASR

---

Under review as a conference paper at ICLR 2023
sequence. Previous studies D'Haro & Banchs (2016); Liao al. (020); Mani al. (2020) have proposed seguuence-to-sqquenee models that decode the target sequence in an autoregressive (AR) manner. WWang al. (2020) added phoneme information to the AR decoder and found that it helps retrieve the correct entity from ASR transeriptions. These autoregressive models achieve SOTA accuracy but incur high latency making them infeasible for online production systems with low- latency constraints. For example. for voice digital assistants the end-to-end latency for a response is the order of milliseconds for high quality user experience. Hence when incorporating such error correction models into the whole system, we need to seriously consider the speed and accuracy trade-off. Autoregressive decoding a big bottleneck as cannot be parallelized during inference, which does not meet the latency buffer allocated to the ASR error correction component in the end- to-end pipeline. Therefore, the critical need of reducing latency brings us the strong motivation to use non autoerrreerree (NAR) models over AR models. Leng et al. (202T) applied NAR sequence generation model with edit alignment to Chinese corpus that achieved comparable WER reduction and is 6 times faster than AR models. However, the performance of this NAR approach has not been tested for English corpus.
In this paper, we propose PATCorrect (Phoneme Augmented Transformer for ASR error Correction) as shown in Figure [ novel NAR based ASR error correction model with edit alignment that is based on both text and phoneme representations of the ASR transcribed sentences. PATCorrect cre- ates inputs for the length tagging predictor by applying a multi-modal fusion approach to combine phoneme representation and text representation into joint feature embeddings. Both encoders (text and phoneme) interact with NAR decoder via encoder-decoder attention mechanism. PATCorrect improves the WER reduction (WERR) to 11.62% compared to the FastCorrect which the SOTA NAR method that solely uses text only representation of the input, with comparable inference latency tens of milliseconds scale. PATCorrect model is robust and scalable to different upstream ASR systems. We use three ASR systems to transcribe two public English corpus datasets, LibriSpeech and Common Voice, respectively to get their erroneous transcriptions as inputs. Experimental evalu- ations demonstrate that applying PATCorrect can consistently improve the transcription WER across different upstream ASR models with varying levels of transcription quality. To demonstrate our per- formance improvement, we benchmark against other ASR error correction models by applying them to the same sets of erroneous transcriptions.
Our contributions are summarized follows:
We propose PATCorrect, a novel model based on the Transformer architecture for NAR ASR correction. This model uses multi-modal fusion approach that augments the traditional input text encoding with an additional phoneme encoder to incorporate pronunciation information, which is one of the key characteristics for spoken utterances.
Through extensive offline evaluations, We demonstrate that PATCorrect outperforms the state-of- the-art NAR ASR error correction model that uses text only modality. For example, PATCorrect improves WERR to 11 11.62% with an inference latency at the same tens of milliseconds scale, while still being about 4.2 6.7x times faster than AR models.
To the best of our knowledge, we are the first to establish that multi-modal fusion is promising direction for improving the accuracy of low latency NAR methods for ASR error correction, and comprehensively study the performance of NAR ASR error correction for English corpus across different ASR systems with varying levels of quality.
2 RELATED WORK
AUTORFGRESSIVE METHODS
The goal of ASR error correction is to convert erroneous source sequences from ASR outputs to target sequences with errors corrected. It can be viewed as Neural Machine Translation (NMT) problem with erroneous sentences as source language, and corrected sentences target language. Therefore, research on ASR error correction started with conventional statistical machine translation methods. Cucu et al. (2013) applied it in domain-specific ASR systems for error correction. Anan- taram et al. (2018) further utilized ontology learning to repair ASR outputs by a 4-step method. Recent NMT methods based on Transformers Vaswani al. (017)); Ng et al. (2019) have become
2

---

Under review as a conference paper at ICLR 2023
increasingly accurate and have inspired applications to ASR error correction Liao et al. 2020); Mani et al. (2020); Hu et al. (2020). Based on the intuition that phonetic information helps with under- standing ASR errors Fang al. (020); Sundararaman al. (202T), Wang al. (2020) found that adding phoneme information for domain-agnostie ASR system could benefit entity retrieval task. Although achieving high accuracy, these encoder-decoder based autoregressive generation models suffer from slow inference speed, error propagation and demand for a large amount of training data.
NON-AUTOREGRESSIVE METHODS
To address these issues in AR models, non-autoregrressvv sequence generation methods in NMT Gu et al. (017)), which aim to speed up the inference AR models while maintaining comparable accuracy, has been popular research topic in recent years. For NAR decoder, the length predictor is crucial as it outputs latent variables to determine the target sequence length for parallel genera- tion. Guu al. 2019) proposed to dynamic insertion/deletion to iteratively refine the generated sequences based on previous predictions. Ghazvininejad et al. (2019) used a conditional masked more efficient iterative parallel decoding. Straight-forward forwwrd adaptation of these NAR methods from machine translation to the ASR error correction problem may even lead increase WER as shown in Leng et al. (202T). Current SOTA method for Chinese corpus er- ror correction Leng et al. (2021) utilized edit alignment with text editing operations (i.e. insertion, deletion, substitution) to train the length predictor and assign each source token with a length tag.
language modeling
3 PATCORRECT FOR ASR ERROR CORRECTION
Predicted
Corrected tokens
tbere're
threse
items
F'
misiin
Tag redictor N x
Add & Norm
Convolution
Joint
MLP
Decoder
N
Add
Multi-modal fusion
Phoneme MHA
Add & Norm
Add Norm
MLP
Phonemb encocer
MLP
Text
Text
encoder
Add & Norm
Add Norm
N
Add & Norm
Self MHA
Self MHA
Self MHA
Text encoding
positional
pasttional encoding
encoding
Phoneme Embeciding
Text Embedding
Text Embeciding
nothing
nothing
The overview of PATCorrect architecture. Starting from the bottom, we input the text se- quence w and phoneme sequence to two encoders separately. The outputs from phoneme and text encoders are combined together by multi-modal fusion operations, and then fed into the tag predic- tor I TagP for adjusting source tokens. We use two encoder-decoder attention layers sequentially in the joint NAR decoder for parallel decoding, to get the target sequences with ASR error correction.
Figure 1:
The goal of ASR error correction is correct the erroneous source tokenized sentence [Wi W2...., Wn to the target error-free sequence z0 = {w1, t02, ..., Cn where the length in- put tokens and the length of output tokens ft be the same or different. In PATCorrect. we add the phoneme sequence p = {PI.P2..... pm.} of the source sentence w to represent the pronunciation information. During training, we first pre-compute the text editing path from source sequence
3

---

Under review as a conference paper at ICLR 2023
target sequence tē in the training set, similar to Le et al. (2021). Then both source sequence and corresponding phoneme sequence are used as imputs to train a tag predictor which generates token- level alignment tags t {t,, f2..... }. For each source token W,, the corresponding alignment tags i consist of 4 possible edit operation types: I; = 1 means keep the token unchanged; 4; = O means -1 means substitute the token with another token; 1, < -1 means add other tokens adjacent to this token, for example, 4 = -5 means add 4 adjacent tokens. Before input into the von-autoregrressvv (parallel) decoder, the source tokens are adjusted based on the corresponding alignment tags in order to match the target sequence. During inference when we do not have ground truth target sequence to compute text editing paths, the tag predictor is used to predict the edit tags for the input source sequence, and then the target sequence is generated according to predicted tags.
delete this token;
3.1 PHONEME AUGMENTED ENCODER
The vanilla Transformer model Vaswani et al. (017) consists of a single encoder and a single de- coder module, each of which is stack of several identical layers. We augment the architecture by adding an additional encoder to provide more information, which conceptually could encode any information modality depending on the application. Here we choose to use phoneme sequence because homophone error is one of the major sources of ASR speech-to-text transcription errors Fang et al. (2020); Sundararaman et al. (202I). The two encoders first encode text and phoneme information separately without sharing model parameters. The stacked encoder layers, consist- ing of multi-head self-attention layers and position- wise fully connected feed-fone forward (MLPs) lay- ers, transform the imput text sequence
and phoneme sequence p into hidden representations H.. (han Jewg, ........ } E Raxdi and Hp (hip Mp2::...lp... E Rmxdi with the output dimension dr. We use the same hidden dimension da for both encoders to simplify the multi-modal fusion operations later. There are two purposes for Hw and Hp: firstly their fused representations are input into tag predictor for predicting the edit alignment corresponding each source token, secondly they both provide inputs to encoder-decoder cross attention mechanism in the joint NAR decoder during parallel decoding.
3.2 MULTI-MODAL FUSION FOR TAG PREDICTOR
The tag predictor Tagli is trained using precomputed ground truth tags with optimal editing align- ment paths from source sequences to target sequences Leng et al. (202I). The text encoder output Hw and phoneme encoder output Hp are combined into H, via multi-modal fusion operations be- fore feeding into the tag predictor get one-dimensional scalar vector t R=x1, that has the same length n as the source sequence. We experiment with different fusion approaches Kiela al.) (2018); Ghazvininejad et al. (2018) combine the text encoder output and phoneme encoder output, and then compare their performances in Sec. 5
Concatenation We concatenate the two encoder outputs H. and Hp together so that H. (bwe N,w2 ...b hps hppa:... E (= By feeding this fused representation to convo- lutional and linear layers, the tag predictor TagP will will output one intermediate vector 7 E (n++m) x that has the same length as the fused input Hg. We then crop 7 by selecting the first n dimensions to get the final tag prediction t E g=x1
Pooling operations Considering that normally n of m for these multi-modal inputs, we first zero-pad the smaller dimension encoder outputs to make them the same length. In this ASR error correction application where the length of phoneme sequences is longer than or equal to the length of the text sequences, i.e. m n. pad Hw E Rd with zeros to create H' E R Then component-wise addition or max pooling can be applied to Hi, and Hp:
Hg = Addition (I', Hp)
(I)
or
H, Max(H Hp)
(2)
where H2 E Rmdd, Similarly, we crop the vector I E R. which is the intermediate output of T(H, by selecting the first dimensions to get the final tag prediction t E R=x!
Cross attention We add cross attention layers with learnable parameters to project the phoneme encoder outputs on the text encoder outputs, which can also handle the embedding length difference
4

---

Under review as a conference paper at ICLR 2023
between the different modalities. We compute the cross attention outputs by taking text encoder output Hw as query, phoneme encoder output Hp as key and value:
(H W (H) (HPWW)
Ha Softmax
Vdh
where W W WY E Rddixdd are the parameter matrices in cross attention layer with multi- ple attention heads, Hs E Raax, the output after additional dropout, residual connection and normalization layers. No further cropping operation is needed since TagP(H,) te R
3.3 NON-AUTOORFGRESSIVE JOINT DECODER
In the conventional AR model, the next token in the output sequence is conditioned on previous predicted tokens to form chain of conditional probabilities:
i+1
PAR w: = II
P(c 10 0)
(4)
These conditional probabilities can be trained autoregressively with cross-entropy loss at each de- coding step. Adapted from Gu et al. (2017). our NAR model utilizes the tag predictor output latent variable to indicate the target length beforehand. So the conditional probability of the target sequence defined as:
1 II
PPAIComeca (1) W. P;
Σ TET
Pragr (ta wp
p; 0) II P(W p:
(S)
where the set of all possible tag sequences that produces the correct editing of target sequences of length îi and wi is the adjusted source inputs based on predicted tags t. The adjusted source sequence w is the actual imput the NAR decoder. Therefore, the training of PATCorrect is equivalent to optimizing the overall maximum likelihood with variational lower bound:
CPAIConent log PrATConeal (10 w. p; 0)
A P(iv wo p. 0) q(t)
log
E t--g
II
pp,
i=1
2
Σ log (ragg(til w. 0 +
log P( w/ p: p:0)
+ Hg(g)
(6)
tng
where H(q) = Eesu (log (t)) is the Shannon entropy. With the optimal edit alignment paths, provide approximate distribution q for the tag sequence with the most likely sample as the expectation. The first term within (.~g(.) trains the tag predictor T TagP, and the second term trains the error correction model. This design enables the parallel decoding for every target token w; the same time.
In the meantime, we include the encoder-decoder attention mechanism for both of encoders in the joint NAR decoder, by sequentially combining them together Wang et al. (2020). After getting the self-attention output from decoder Ha E Ráxdd, with causal mask removed to enable parallel calculation similar to Gu et al. (201)). we calculate the text and phoneme encoder-decoder attention by first using Ha as query, text encoder output H. as key and value:
HH W ) H,,.. WKYT
Zn = Softmax/
[H W.)
ddh.
and then using the output Z., as query, phoneme encoder output Hp as key and value:
7, = Softmax (Z (Zn WHH (H,W))
(8)
Vdh
p. W.K. wy) E Rdaxd xd. are the parameter matrices in the text and phoneme encoder-decoder multi-head attention layers, respectively. The remaining of the decoder layers are the same as the vanilla Transformer model where Z, E R &xd, will be input into fully- connected MLPss with residual connection and layer normalization.
where (WO, W W {W?,
5

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
4.1 DATASETS AND ASR MODELS
We create training data specifically designed for correcting ASR errors because ASR systems are ac- curate for most tokens and source sentences usually aligned monotonically with target sentences unlike the shuffle error in NMT. For the ASR transcription dataset S multiple ASR systems TRTiR used to transcribe public English audio corpus A = St where the audio clips C have corresponding ground truths from human transcriptions Strans The transcriptions of C from glrans,k
(C) contain actual ASR errors, which leads to the ASR transcription -==1 (S It oz = U Uk=1 (A,(CC), Stan), ). We combine two public English datasets, LibriSpeech Panayotov et al. 2015 and Common Voice v9.0 v9.0/Ardila et al. 2019)) together to get corpus A.
ASR systems Sf
Ax(C)
dataset gurans = -
In order to test different ASR systems with different architecture and performances, we choose pre-trained ASR systems [Az] implemented in NeMo Kuchaiev al. (2019): The Convolution- augmented Transformer Conformer Gulati et al. 2020 trained on several thousands hours of English speech with top-tier performance; The Convolution-
Jasper Li et al. (2019 2 trained on 7,057 hours of audio samples with above average performance; A light-weight 5x5 QuartzNet Kriman et al. 2020 trained on 960 hours LibriSpeech dataset with subpar performance. All ASR systems are trained with the Connectionist Temporal Classification (CTC) loss. We use the default splits in LibriSpeech and Common Voice with transcriptions from 3 ASR systems together to compose the dataset Strans as shown in Table which in total has more than 3.5 million sentences pairs in training split. Note that LibriSpeech has been included in the training of all 3 ASR systems, so we use DEV and TEST splits from Common Voice as benchmarks accuracy evaluations later.
Table 1: ASR transcription dataset statistics and ASR original WER
LibriSpeech DEV
Common Voice
TRAIN
TEST
TRAIN
DEV
TEST
# of sents
281,241
5,567
5,559
890,107
16,331
16,318
Avg. words/sent
33.4
18.9
18.9
10.3
9.8
9.4
Conformer WER
1.58
3.21
3.27
7.62
9.84
10.03
Jasper WER
2.13
6.87
6.90
13.63
18.91
21.02
QuartzNet WER
1.97
10.37
11.01
36.42
47.90
52.85
4.2 EVALUATION METRICS
The performance of ASR етrо correction models should be evaluated in two aspects: accuracy and speed. Speed is measured by the latency of the whole inference process including encoding and decoding for different methods. For accuracy, (WERR) Fo.s. Fo.25. and Correction. Similar to Leng et al. 2021), WER is defined as total edit distance between source and target divided by the total number of words in target sequence, and WERR is defined as the percentage of improvement in WER. For error detection ability, Precision measures how many actual error tokens are edited among all of the edited tokens, i.e., how many of them actually need to be corrected; Recall measures how many actual error tokens edited among all of the error tokens. In this application of ASR error correction, unnecessary edits may even lead to WER increase because original ASR outputs are mostly correct. So to put more weight on Precision, we use Fb.s and Fu 25 as overall measurements for error detection ability as in Liao et al. 2020): Omelianchuk al. 2020); Rothe al. (202I). For error correction ability, Correction,
use Word Error Rate (WER), WER Reduction
en conformer ctc large "htps://catalog.n www..nnnii...iiooooooooo www.p//wwwwg.oocooooe////eeoeaat en ja
hhtps://catalog.ngg/ ngc. wwwpai/wwww....../eeee//eeeeeeea
"QuartzNet5x51.S- S-En:: https:/catallg..ng..n..
total edit distances total number of substitutions + insertions deletions

---

Under review as a conference paper at ICLR 2023
defined as (# of correctly edited error tokens)/ (# of edited enror tokens), measures the percentage of edited error tokens that match the ground truth.
4.3 MODEL CONFIGURATIONS
PATCorrect In our PATCorrect architecture, we use 6-layer text encoder, 6-layer phoneme en- coder and 6-layer joint decoder with the hidden model dimension d, 512, and MIP dimension dMLP = 2048. We use 8 attention heads for self-attention, encoder-decoder attention and cross atten- tion respectively. In the cross attention setup for fusing two encoder outputs, we use consecutive modules of cross attention with dropout, residual connection and layer normalization. For tag pre- dictor TagP. we apply 5 layers of convolutional modules which consists of 1-D Convolution layer with kernel size equals 3, ReLU activation, layer normalization and dropout, followed by 2 layers of MLPs generate mi-dimensional vector which is the output from tag predictor TagP. We use MSE loss to train TagP to predict the length adjustment for each source token.
Baseline models We compare our model with both AR Transformer and popular NAR methods. For AR Transformer, we use the standard vanilla Transformer architecture with 6-layer encoder and 6-layer decoder and the same hidden size d3 = 512. For NAR methods from NMT, we use the default training hyperparameters from Mask Predict (CMLM) (CMLM) Ghazvininejad et al. 2019) and Lev- enshtein Transformer (LevT) ) Gu et al. (2019 5 For SOTA NAR method for ASR error correction adapt the same 6-layer encoder-decoder architecture and the same architecture for its length predictor.
problem FastCorrect Leng et al. (2021
Training and inference details All models are implemented using Fairseq Ott et al. 2019), and trained using 4 NVIDIA Tesla V100 GPUs with maximum batch token size of 5000 and la- bel smoothed cross entropy loss function. They trained from scratch using the ASR transcription dataset Strans for 30 epochs. Source and target sentences are tokenized using sentencepiece Kudo & Richardson 2018)) and the phoneme sequences are generated by English grapheme phoneme conversion using the CMU pronouncing dictionary an independent post-processing step of the 1- best ASR hypothesis as model inputs. We use Adam optimizer Kingma & Bal (2014) and inverse square root for learning rate scheduling starting from 5e During inference, we set the test batch size as to simulate automated machine learning system environment with output from the up- stream ASR system, and all NAR methods have the same max decoding iteration of 1. Different hardware conditions are tested including single NVIDIA Tesla V100 GPU, 8 CPUs and 4 CPUs with Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz.
RESULTS
5.1 ACCURACY
We compare the WER and WERR using different error correction models for all three ASR systems and their total combined transcriptions on Common Voice DEV and TEST splits, as shown in Table and 3 The lower WER and higher WERR, the better model accuracy. Results show: 1) In total, the proposed PATCorrect model beats the SOTA FastCorrect method by improving the TEST set WERR from 9.46 to 1.62. which is more than 20% relative improvement; 2) Our PATCorrect model outperforms other NAR methods robustly when dealing with all three ASR system transcriptions. Straihht-forwat adaptations of NMT NAR methods like CMLM and LevT may even introduce more WER for some well-performing ASR systems like Conformer and Jasper; 3) Among all of the multi-modal fusion operations experimented, cross attention performs best across almost all datasets.
5.2 SPEED
We test inference speed both GPU and CPUs shown in Table 4 The inference latency LibriSpeech (LS) longer than that Common Voice (CV) because LibriSpeech has twice average
http:///we........oooooooooooooe translation hhapy//iitt....mmoooooooooaaaaae https://githuh.oo///yyyyyynn

---

Under review as a conference paper at ICLR 2023
Table
WER (%) & using different error correction models
Conformer
Jasper
Quartz.Net
Total
Models
DEV
TEST
DEV
TEST
DEV
TEST
DEV
TEST
No error correction
9.84
10.03
18.91
21.02
47.90
52.85
25.55
27.96
AR Transformer
8.56
9.26
15.95
18.56
33.84
40.64
19.45
22.82
CMLM 10.43
10.72
19.25
21.37
42.09
47.39
23.92
26.49
LevT
10.05
10.37
18.69
20.74
40.82
45.98
23.19
25.70
FastCorrect
9.25
9.88
17.45
19.96
39.69
46.11
22.13
25.32
PATCorrect(cu)
9.22
9.78
17.45
19.93
39.61
46.00
22.09
25.24
PATCorrect(
9.26
9.83
17.50
19.96
39.67
46.04
22.14
25.28
PATComeet (
9.20
9.79
17.34
19.82
39.25
45.60
21.93
25.07
PATCorrect (cross (cross_atten) _atten)
9.15
9.84
17.28
19.80
38.27
44.50
21.57
24.72
Table 3:
WERR (%) f using different error correction models
Conformer
Jasper
QuartzNet
Total
Models
DEV
TEST
DEV
TEST
DEV
TEST
DEV
TEST
AR Transformer
13.07
7.65
15.64
11.70
29.34
23.10
23.87
18.40
CMLM
-6.00
-6.89
-1.80
-1.67
12.12
10.32
6.36
5.26
LevT
-2.15
-3.41
1.15
1.35
14.77
12.99
9.24
8.11
FastCorrect
5.98
1.41
7.68
5.05
17.13
12.74
13.37
9.46
PATCorrect(en))
6.29
2.44
7.73
5.17
17.31
12.95
13.53
9.75
(ATCorrect(add)
5,90
1.94
7.45
5.04
17.18
12.88
13.33
9.61
(ATCorrec((maz)
6.50
2.32
8.28
5.71
18.05
13.71
14.16
10.35
AATCorrect(cross aaten)
7.01
1.81
8.61
5.80
20.11
15.79
15.59
11.62
words per sentence shown in Table Consistent with the observation from Gu et al. (2017). AR model shows a linear latency trend with decoding lengths, while the latency of the NAR methods only slightly increases. PATCorrect achieves inference latency which is comparable with other NAR models, especially on GPU hardware it is only slower than FastCorrect, while still being about 4.2 6.7x times faster than AR models on Common Voice and LibriSpeech datasets.
Table 4: Inference latency | using different hardwares with unit ms/sentence
Single GPU
8 CPUs
4x
Models
LS
CV
LS
CV
LS
CV
AR Transformer
246.74
141.00
306.40
149.79
366.54
186.89
CMLM
39.49
35.54
47.96
43.78
55.15
47.90
LevT
52.61
47.86
49.89
45.68
60.27
55.33
FastCorrect
22.25
20.81
29.28
23.77
36.02
29.14
(ATCoorea((rrosssaate)))
36.92
33.71
52.47
41.89
66.97
49.97
5.3 SENSITIVITY ANALYSIS
To further investigate the performance improvements PATCorrect, we conduct sensitivity analysis by comparing the еrror detection ability and error correction ability for different models with ASR system transcriptions in total. Table 5 P, Rt C denote Precision, Recall, Correction respectively. Results show that our PATCorrect not only has the highest Fo.5 and Fo.25 score with great Precision that is comparable to AR model, but also has better ability to edit error tokens to the correct targets indicated by higher Correction. As shown in the examples in Appendix Table 7 our PATCorrect's 's design of adding a phoneme information helps our model identify and correct the sound-alike ASR errors that previous method couldn't 't correct with only text information.
8

---

Under review as a conference paper at ICLR 2023
Table Sensitivity metrics for error correction models
DEV
TEST
Models
P
Fo.5
F0.25
C
P
R
Fo.5
F0.25
AR
91.50
66.72
85.17
89.54
44.53
90.89
65.33
84.30
88.85
37.81
Transformer
CMLM
83.39
61.70
77.92
81.70
31.43
84.87
61.45
78.86
83.01
27.37
LevT
85.08
61.95
79.17
83.26
32.53
86.49
61.75
80.07
84.49
28.42
FastCorrect
90.25
61.64
82.58
87.85
32.66
89.53
60.90
81.83
87.12
27.70
PATCorrect
91.42
60.28
82.86
88.72
34.45
90.27
59.64
81.86
87.62
29.50
(cross_allen)
5.4 ABLATION STUDY
We perform ablation study to understand the effectiveness of different components PATCorrect by removing component while retaining the others. No phoneme input for TagP means that we only use text information as input for predicting token tags while still using phoneme encoder-decoder attention in NAR decoder. No phoneme attention means that we remove phoneme encoder-decoder attention from the NAR decoder and only use text encoder-decodee attention, while still using cross- attention to fuse the and phoneme encoder outputs for Tagl. We also increase the amount of pre-training data by using synthetic dataset Sayoth for data augmentation to pretrain the model for 20 epochs. We crawl 50M sentences from English Wiki page: and add random editing operations like deletion, insertion, substitution with homophone dictionary to produce erroneous sentences paired with the original correct texts, that mimics the ASR errors with a simulated error distribution. The results in Table O with WER and WERR, weiuht averaged from all 3 ASR systems, show that adding phoneme information in TagP and NAR decoder lead to better WER and WERR. Our PATCorrect, using phoneme information in both tag predictor and NAR decoder, yields the best results. In addition, using synthetic dataset for pretraining can also boost the model accuracy to further reduce WER.
Table 6: Ablation study for PATCorrect model
DEV
TEST
Models
WER 4
WERR 1
WER 4
WERR f
No eтrоr correction
25.55
27.96
FastCorrect
22.13
13.37
25.32
9.46
No phoneme input for TagP
21.88
14.35
25.13
10.14
No phoneme attention in decoder
21.90
14.30
25.06
10.39
PATCCorrec(croos atten)
21.57
15.59
24.72
11.62
PXTCorrect(rooss_ateee Ssynth
21.12
17.35
24.33
13.00
CONCLUSIONS
We propose PATCorrect, novel NAR phoneme-auggmented Transformer-bas based model with robust performance on different upstream ASR systems with varying speech-to-text transcription quality, serve as independent post-processing text editing component for reducing the errors in ASR transcriptions. Our model outperforms state-of-the-arr NAR ASR eгror correction models, and still 4.2 6.7x times faster than AR models, which makes it a great fit industrial scale text editing method to refine ASR transcriptions. Our study establishes that multi-modal fusion is promising direction for improving the accuracy of low latency NAR methods for ASR error correction. Fu- ture explorations including adding context information for source sequences, and using knowledge distillation with AR model as teacher Zhou al. (2019) to improve the performance PATCorrect.

---

Under review as a conference paper at ICLR 2023
REFERENCES
C Anantaram, Amit Sangroya. Mrinal Rawat, and Aishwarya Chhabra. Repairing asr output by artificial development and ontology based learning. In IJCAI, pp. 5799-5801, 2018.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively-mulilingual speech corpus. arXiv preprint arXiv:1912.066700 2019.
Horia Cucu, Andi Buzo, Laurent Besacier, and Comeliu Burileanu. Statistical enror correction methods for domain-specific asr systems. In International Conference on Statistical Language and Speech Processing, pp. 83-92. Springer, 2013.
Luis Fernando D' O'Haro and Rafael E Banchs. Automatic correction of asr outputs by using machine translation. In Interspeech, volume 2016, pp. 1469-3347, 2016.
Anjie Fang, Simone Filice, Nut Limsopatham, and Oleg Rokhlenko. Using phoneme representations to build predictive models robust to asr errors. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 699-708, 2020.
Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. A knowledge-groundee neural conversation model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. arXiv preprint (x0x:09994.09322 2019.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non-autoreerressvv neural machine translation. arXiv preprint 1.Xvv11111.02211. 2017.
Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. Advances in Neural Infor-
mation Processing Systems, 32, 2019.
Anmol Gulati, James Qin, Chung Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: -agulliioo-aummneeee transformer for speech recognition. arXiv preprint arXhv:2005.08100, 2020.
Ke Hu, Tara N Sainath, Ruoming Pang. and Rohit Prabhavalkar. Deliberation model based two- pass end-to-end speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7799-7803. IEEE, 2020.
Douwe Kiela, Edouard Grave, Armand Joulin, and Tomas Mikolov. Efficient large-scale multi- modal classification. In Proceedings the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Samuel Kriman, Stanislav Beliaev, Boris Ginsburg, Jocelyn Huang, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, and Yang Zhang. Quartznet: Deep automatic speech recog- nition with 1d time-channel separable convolutions. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6124 6128. IEEE, 2020.
Oleksii Kuchaiev, Jason Li, Huyen Nguyen, Oleksii Hrinchuk, Ryan Leary, Boris Ginsburg, Samuel Kriman, Stanislav Beliaev, Vitaly Lavrukhin, Jack Cook, al. Nemo: toolkit for building ai applications using neural modules. arXiv preprint arXiv: arXiv: /99999999999........ 2019.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06222 2018.
Yichong Leng, Xu Tan, Linchen Zhu, Jin Xu, Renqian Luo, Linquan Liu, Tao Qin, Xiangyang Li, Edward Lin, and Tie- Yan Liu. Fastcorrect: Fast error correction with edit alignment for automatic speech recognition. Advances in Neural Information Processing Systems,
2021.
10

---

Under review as a conference paper at ICLR 2023
Jason Li, Vitaly Lavrukhin, Boris Ginsburg, Ryan Leary, Oleksii Kuchaiev, Jonathan M Cohen, Huyen Nguyen, and Ravi Teja Gadde. Jasper: An end-to-end convolutional neural acoustic model. arXiv preprint arXiv: 190.0280, 2019.
Junwei Liao, Sefik Emre Eskimez, Liyang Lu, Yu Shi, Ming Gong, Linjun Shou, Hong Qu, and Michael Zeng. Improving readability for automatic speech recognition transcription. arXiv preprint arXiv:2004.044333 2020.
Anirudh Mani, Shruti Palaskar, Nimshi Venkat Meripo, Sandeep Konam, and Florian Metze. Asr error correction and domain adaptation using machine translation. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6344-6348. IEEE, 2020.
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. Facebook fair's wmt19 news translation task submission. arXiv preprint urXiv:1907006111 2019.
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chemodub, and Oleksandr Skurzhanskyi. Gector cectoo-grammatical error correction: tag, not rewrite. arXiv preprint arXiv:2005. 12592, 2020.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based public domain audio books. In 20/5 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210, IEEE, 2015.
Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. A simple recipe for multilingual grammatical error correction. arXiv preprint urXiv:2106.03880 2021.
Mukuntha Narayanan Sundararaman, Ayush Kumar, and Jithendra Vepa. Phoneme-bert: Joint lan- guage modelling of phoneme sequence and asr transcript. arXiv preprint arXIvv2102.000004 2021.
Zoltán Tüske, George Saon, and Brian Kingsbury. On the limit of english conversational speech recognition. arXiv preprint vrXiv:2105.00982 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Haoyu Wang. Shuyan Dong, Yue Liu, James Logan, Ashish Kumar Agrawal, and Yang Liu. Asr error correction with augmented transformer for entity retrieval. In Interspeech, pp. 1550-1554, 2020.
Chunting Zhou, Graham Neubig, and Jiatao Gu. Understanding knowledge distillation in autoregressive machine translation. arXiv preprint arXiv:1911.02227, 2019.

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
We show some examples in Common Voice TEST split with ground truths, QuartzNet transcriptions, FastCorrect and PATCorrect corrections.
Table 7: Examples for enror correction sentences
Ground truths
ASR transcriptions
FastCorrect corrections
PATCorrect corrections
don't blame you
don't lame you
don't lame you
don't blame you areabased visualiza- tions have existed for decades
areabased visu- alizations have existed for decades
arcobased visualiza- tions have existed for decates
a realbased visualiza- tions have existed for decays
a blond woman is singing on stage
a blond woman is singing on states
blond woman is singing sticks
blond woman is singing on stage
a white bike is leaning against post
a white wike is lean- ing against the post
a white white leaning against the post
white bike leaning against post
he attended carnegie mellon university
he attended canegy melon niversity
he attended carnegie milan university
he attended carnegie mellon university
make sure you get a doctor's note
make sure you get doctor's nowte
make sure you a doctor's know
make sure you get doctor's note
he was the prime suspect out of thir- teen
he was the prime sus- pect out of thirte
he was the prime sus- pect out of the team
he was the prime sus- pect out of thirteen
we need to run
wanet to run
what it to run
we need to run she was silent for mo- ment or two
she was silent for moment or two
she was silent for mo- she was silent for moment or to it was one of the furest private com- mercial brod gaster in the nidderlans
ment or too
was one of the first private commercial broad- caster in the netherlands
was one of the first private commercial broadcaster in the midlands
it was one of the first private commercial broadcaster in the netherlands
they are available in shades of white pink purple and blue
they are available in shades white pak purple and blue
they available in shades of white pig purple and blue
they available in shades of white pink purple and blue
the bloom of the rose lasts a few days
the bloom of the rors lasts few days
the bloom of the rows lasts few days
the bloom of the rose lasts a few days
man in blue shirt sitting at a bus stop
man in blue shore tos sitting at a busttup
man in blue shirt is sitting at a bus
man in blue shirt is sitting at bus stop
12


Reviews:
[{"review_id": "gBLeWr2nJZ", "reviewer": "Reviewer_RfZS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new non-autoregressive model that incorporates the pronunciation information with text information for the task of ASR error correction. The proposed model uses a transformer architecture for two encoders and a decoder, it adopts three different fusion approaches to combine the representations from phoneme encoder and text encoder for the target length prediction. The decoder computes the cross attention with both encoders sequentially and predicts the target words. ", "summary_of_the_review": "This paper proposes a network of NAR Transformer with a phoneme encoder. Compared with FastCorrect, the improvement is limited, but it requires more latency than FastCorrect. Besides, the idea of incorporating phoneme information to gain better ability of error correction is useful but not very creative, since many text correction approaches have been proposed in recent years with similar ideas.", "strength_and_weaknesses": "Strength:\n\n1.The idea of phoneme fusion and cross attention strategy is simple yet effective. The model outperforms FastCorrect on English datasets, and achieves SOTA results on English corpus.\n\n2. The paper compares three feature fusion approaches, and shows that cross attention is effective for text and phoneme features.\n\n3. Experimental results are based on comparisons on 3 English ASR systems to show the robustness of the proposed approaches.\n\nWeakness：\n1. A Phoneme MHA module is added behind the original Text MHA module in the Transformer decoder block, but the ablation study shows that without this phoneme attention, the model performance roughly remains the same as adding it. So with more calculation on the phoneme attention, the model barely gets any improvement from it.\n2. The improvements on some datasets are minor. All of the fusion approaches of combining the phoneme embedding and text embedding don’t consider the relationship between each word and its corresponding pronunciation, which may be important for the words error correction.\n3. The novelty of proposed framework is not significant. The architecture does not seem that there is too much difference with FastCorrect except the phoneme embedding, and two extra cross attention. Besides that, model architecture and train process are almost the same.\n4. Phoneme feature is commonly used in error correction [1, 2], and from the ablation experiments we see a lot of contributions are from phoneme feature. To my knowledge cross-attention feature fusion can be seen in Image field previously [3].\n\n[1] Wang et al., ASR Error Correction with Augmented Transformer for Entity Retrieval. Interspeech 2020.\n\n[2] Fang et al., Non-Autoregressive Chinese ASR Error Correction with Phonological Training. NAACL 2022.\n\n[3] Bai et al., TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers. CVPR 2022.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dkI4nGcKi7", "reviewer": "Reviewer_XMid", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "PATCorrect is a non-autoregressive (NAR) error correction module.  PATCorrect is distinguished by using aligned phoneme and word inputs for making correction decisions.  Evaluation across a number of input ASR models demonstrates that PATCorrect outperforms FastCorrect, a similar NAR correction model which does not use phone information.", "summary_of_the_review": "The central contribution of the work is strong, and clearly described.  The evaluations themselves are good.  However, the presentation could be more tempered to more accurately demonstrate the strengths and weaknesses of the approach.  A stronger motivation for the use of NAR error correction will also provide additional context for the contribution of the work.", "strength_and_weaknesses": "Strength\n* The use of phonetic information in error correction is well motivated.\n* The architecture is well described and well motivated.\n\nWeakness\n* The improvements offered by PATCorrect are somewhat modest.  There is a small, but consistent improvement over FastCorrect.  But the performance is still substantially worse than AR Transformer. \n* The reporting of performance is, at some points, strained.  1) The decision to use F_0.25 instead of a more common detection metric of F_1 is surprising, but the authors gesture toward the bias toward Precision.  However, this decision is less convincing when observing that FactCorrect has a better F_1 score than PATCorrect, though PATCorrect has a higher F_0.25 score.   2) The reported improvement of 20% in the abstract and introduction is a relative improvement to the relative improvement to WER (the primary metric for ASR).  This is not clear in the abstract.  The absolute WER difference between FastCorrect and PATCorrect is between 1 and 2%. WERR (relative improvement to WER) may be a more relevant measure for error correction, however, since this is already a relative measure, absolute difference to WERR are a more interpretable description of performance differences.  The underlying work is strong, but these presentation decisions leave the impression that the work is presented in an overly optimistic manner.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SJ1PbQlzX9", "reviewer": "Reviewer_9fDp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The task is automatic speech recognition (ASR). The paper proposed PATCorrect, a new correction model which operates on the output of another speech recognition model and tries to improve the output by correcting errors.\n\nThe proposed model works in a non-autoregressive way and thus can run in parallel, efficiently on hardware like GPUs.\n\nIt uses three different pretrained models to test the correction model on. The best model is a Conformer, which gets 10.03% WER on CommonVoice Test without correction, and the best PATCorrect correction gets down to 9.78% WER. The other models are very weak in comparison, but then the relative improvements are larger.", "summary_of_the_review": "There are too many weaknesses, as explained above. This needs more work.\n\nIn general, this paper seems more appropriate for a speech conference (Interspeech etc)? While in principle it might be applicable to other tasks, I think this should be tested, and I think this is necessary for publication on ICLR. The novelty and scope is otherwise clearly not high enough. Otherwise, even for speech conferences, I think the quality is not high enough yet, and this needs to address the weaknesses I explained before.\n", "strength_and_weaknesses": "Strengths:\n\n- An interesting novel model for error correction.\n- I think the use of phoneme inputs is novel.\n\nWeaknesses:\n\n- No comparison to a standard language model fusion? See below.\n- The code is not published?\n- The analysis and ablations study is too short.\n- Ablation studies only with weak model.\n- Only tested on speech recognition.\n- Many aspects are unclear. See below.\n- Studies are on Common Voice, but LibriSpeech would be better, as this is what most people know much better.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject