Venue: NAACL 2022

# Non-Autoregressive Chinese ASR Error Correction with Phonological Training

Zheng Fang1,2, Ruiqing Zhang3, Zhongjun He3, Hua Wu3, Yanan Cao1,2

###### Abstract

Automatic Speech Recognition (ASR) is an efficient and widely used input method that transcribes speech signals into text. As the errors introduced by ASR systems will impair the performance of downstream tasks, we introduce a post-processing error correction method, PhVEC, to correct errors in text space. For the errors in ASR result, existing works mainly focus on fixed-length corrections, modifying each wrong token to a correct one (one-to-one correction), but rarely consider the _variable-length_ correction (one-to-many or many-to-one correction). In this paper, we propose an efficient non-autoregressive (NAR) method for Chinese ASR error correction for both cases. Instead of conventionally predicting the sentence length in NAR methods, we propose a novel approach that uses phonological tokens to extend the source sentence for _variable-length_ correction, enabling our model to generate phonetically similar corrections. Experimental results on datasets of different domains show that our method achieves significant improvement in word error rate reduction and speeds up the inference by 6.2 times compared with the autoregressive model.

+
Footnote †: This work was conducted at Baidu. Corresponding author: Ruiqing Zhang. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5907 - 5917

**Keywords** phrases: words, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases, phrases,, phrases, phrases, phrases,, phrases, phrases, phrases,, phrases, phrases,, phrases, phrases,, phrases, phrases,, phrases,, phrases, phrases,, phrases,, phrases,, phrases,, phrases,, phrases,, phrases,, phrases,, phrases,, phrases,, phrases,, phrases,, phrases,, phrases,, phrases,, phrases,, phrases,, phrases,, phrases,,, phrases,, phrases,,, phrases,, phrases,,, phrases,, phrases,,, phrases,,, phrases,,, phrases,,, phrases,,, phrases,,, phrases,,, phrases,,, phrases,,, phrases,,, phrases,,, phrases,,, phrases,,, phrases,,,, phrases,,, phrases,,,, phrases,,,, phrases,,,, phrases,,,, phrases,,,, phrases,,,, phrases,,,, phrases,,,, phrases,,,, phrases,,,, phrases,,,, phrases,,,, phrases,,,, phrases,,,,, phrases,,,, phrases,,,,, phrases,,,,, phrases,,,,, phrases,,,,, phrases,,,,, phrases,,,,, phrases,,,,, phrases,,,,, phrases,,,,, phrases,,,,, phrases,,,,, phrases,,,,, phrases,,,,,, phrases,,,,,, phrases,,,,,, phrases,,,,,, phrases,,,,,, phrases,,,,,, phrases,,,,,, phrases,,,,,, phrases,,,,,, phrases,,,,,, phrases,,,,,, phrases,,,,,,, phrases,,,,,,, phrases,,,,,,, phrases,,,,,,, phrases,,,,,,, phrases,,,,,,, phrases,,,,,,, phrases,,,,,,, phrases,,,,,,, phrases,,,,,,,, phrases,,,,,,,, phrases,,,,,,,, phrases,,,,,,,, phrases,,,,,,,, phrases,,,,,,,,, phrases,,,,,,,,, phrases,,,,,,,,, phrases,,,,,,,,, phrases,,,,,,,,,, phrases,,,,,,,,,, phrases,,,,,,,,,,, phrases,,,,,,,,,,,, phrases

aligned to each source token, then up-sampling or dropping some source tokens accordingly for parallel correction. However, the method is fragile because incorrect length prediction will distort the meaning of the sentence and may directly lead to false prediction. See Figure 1 (c) for example, the length predictor produces a wrong alignment (0 2 1 1), indicating that the decoder will drop the first source character and generate two target characters for the second source character. According to the alignment, the intermediate result turns to be a meaningless sentence ("\(\text{``}\text{''}\text{''}\)", means "watch such sad") and leads to a wrong correction.

To address the issues mentioned above, we propose a novel NAR model named PhVEC (Phonology-based Variable-length Error Correction). The model incorporates the phonological features of error characters to enable _variable-length_ prediction. Concretely, PhVEC contains a detection network and a correction network based on a pre-trained language model Devlin et al. (2019); Sun et al. (2020). The detection network predicts the correctness of each token, and the correction network generates correction result. Instead of using a length predictor to predict the fertility for each source token, we insert the phonological tokens Chinese Pinyin1) to the source sequence as placeholders after each detected erroneous character. During prediction, each source token (either the character token or pinyin token) can generate zero or one target token. With the guidance of the phonological token, the model will generate characters with similar pronunciations. As Figure 1 (d) shows, the pinyin token "\(b\)" produces target character ("\(\text{``}\text{''}\)", \(bu\)) with the same consonant letter, and "\(a\)\(o\)" produces target character ("\(\text{``}\text{''}\)", \(yao\)) with the same vowel letter. We delete the repeated characters in the final sequence.

Footnote 1: In this paper, we ignore the tone of pinyin, and the pinyin of each Chinese character can be represented by one to six English letters.

We evaluate our methods on multiple datasets with varying degrees of ASR word error rates. Experimental results show that our PhVEC obtains significant improvement (over 10% word error rate reduction) on standard benchmarks compared with existing NAR-based ASR error correction methods at comparable speed. Even compared with the AR baseline Transformer and BART Lewis et al. (2020) models, PhVEC can still have an 8.55% and 2.39% word error rate reduction while keeping a 6.2x speed-up.

## 2 Method

Chinese ASR error correction can be formalized as the following task. Given a speech recognition sequence \(X=(x_{1},x_{2},...,x_{n})\) of \(n\) Chinese characters, the goal is to correct it into another sequence of \(m\) characters \(Y=(y_{1},y_{2},...,y_{m})\). Note that the target sequence length \(m\) does not have to be equal to the source sequence length \(n\). There exist three types of ASR errors in transforming from \(X\) to \(Y\): substitution, deletion, and insertion.

As illustrated in Figure 1(d), our proposed ASR

Figure 1: An example of Chinese ASR Error Correction. The source speech “_ni bu yao nan guo ”_ (“Don’t be sad”) is incorrectly recognized to “_ni biao nan guo_ ” (“Your watch is sad”). (a) The Seq2Seq model generates a fluent but incorrect sentence because it neglects the phonetic similarity. (b) The sequence tagging model with phonological features correctly detects the character with error (‘K’’ means keep unchanged and ‘C’’ means to be correct), but because it only supports equal-length prediction, it fails to generate a fluent correction with equal length as the input. (c) The NAR model with a length predictor incorrectly predicts the number of target tokens corresponding to the source character “_ni_” as 0, which should be 1. Accordingly, it removes the character and generates an incorrect result. (d) The proposed PhVEC first detects problematic tokens, then leverages their phonological features to generate a _variable-length_ intermediate result, and outputs the final result by reducing the repeated characters.

 

error correction model is composed of an error detector and an error corrector, both perform a NAR tagging. The error detector takes \(X\) as input to predict the correctness of each token. The error corrector takes the combination of the source tokens and the phonological features of problematic tokens as input for correction. During error correction, each token of the extended input can be tagged to a Chinese character or a blank token "\(\epsilon\)", so that the model can support _variable-length_ correction. Since different pinyin letters may generate the same character, we eliminate continuous repetitive characters in the final result. In this paper, we use Chinese pinyin as the phonological feature. The learning of PhVEC is conducted end-to-end, with the error detector and corrector optimized jointly.

### Error Detection Network

The goal of the error detector is to check whether a character \(x_{i}\) (\(1\leq i\leq n\)) is correct or not. We model this task as sequence labeling. We build a sequential binary classifier and use class 1 and 0 to label the problematic characters and correct characters, respectively. The ground-truth detection label \(C=(c_{1},c_{2},...,c_{n})\) is pre-calculated by matching \(X\) and \(Y\) with edit distance, in which \(c_{i}\in\{0,1\}\). The prediction result of the error detector is represented by a sequence \(C^{\prime}=(c^{\prime}_{1},c^{\prime}_{2},...,c^{\prime}_{n})\), and we use \(p_{i}\) to denote the probability of token \(x_{i}\) being predicted to class 1, which can be formalized as follows:

\[p_{i}=p(c^{\prime}_{i}=1|X)=softmax(f_{dec}(E(\mathbf{e_{w}}))) \tag{1}\]

where \(\mathbf{e_{w}}=(e_{x_{1}},e_{x_{2}},...,e_{x_{n}})\) is the token embedding of \(X\), \(E\) is a Transformer-based encoder and \(f_{dec}\) is a fully-connected layer that maps the sentence representation to a binary sequence. To train the model, we adopt the following cross entropy loss function:

\[\mathcal{L}_{dec}=-\frac{1}{n}\sum_{i}\left[c_{i}\ln p_{i}+(1-c_{i})\ln(1-p_{i })\right] \tag{2}\]

### Error Correction Network

After identifying the errors, we introduce the pinyin features of the incorrect characters and feed them into the correction network. Concretely, we use the tool PyPinyin2 to generate pinyin for each erroneous Chinese character and insert pinyin tokens after the problematic token. The original sentence \(X=(x_{1},x_{2},...,x_{n})\) is thus rewritten as \(X_{cor}=(x^{\prime}_{1},x^{\prime}_{2},....,x^{\prime}_{t})\), where \(t\geq n\), and \(t-n\) is the number of pinyin tokens.

Footnote 2: [https://github.com/mozillazg/python-pinyin](https://github.com/mozillazg/python-pinyin)

Given \(X_{cor}\), the error corrector performs sequence labeling to predict the correction result \(Y^{\prime}=(y^{\prime}_{1},y^{\prime}_{2},...,y^{\prime}_{t})\) as follows:

\[p(y^{\prime}_{i}=\mathcal{V}_{j}|X_{cor})= softmax(f_{crt}E(\mathbf{e_{cor}})) \tag{3}\]

where \(p(y^{\prime}_{i}=\mathcal{V}_{j}|X_{cor})\) is the conditional probability that \(x^{\prime}_{i}\) is corrected to \(\mathcal{V}_{j}\), the token with index \(j\) in the vocabulary \(\mathcal{V}\). \(\mathbf{e_{cor}}\) is the token embedding of \(X_{cor}\) and \(f_{cnt}\) is a fully-connected layer that maps the hidden states of source tokens to their predicted logit vector of length \(|\mathcal{V}|\). Note that, the parameters of the token embedding, the encoder \(E\) and the correction network \(f_{ort}\) are initialized by pre-trained language models Devlin et al. (2019); Sun et al. (2020). We share the pre-training model parameters in the error detector and corrector to encode the Chinese characters and pinyin tokens in a shared space, which not only reduces the model size, but also makes the model yield better semantic representation.

The learning objective of the error corrector is to correct \(X_{cor}\) to the golden correction \(Y\). However, it is not an easy task because the lengths of \(X_{cor}\) and \(Y\) may not be the same. Therefore, we rewrite \(Y\) to \(\hat{Y}=(\hat{y}_{1},\hat{y}_{2},...,\hat{y}_{t})\) which has the same length as \(X_{cor}\). Concretely, \(\hat{Y}\) is constructed from \(Y\) with some tokens repeated and inserted according to pronunciation alignment. We compare the pinyin of the incorrect tokens of \(X\) with their

Figure 2: An illustration of generating \(X_{cor}\) and \(\hat{Y}\) from the training sample (\(X,Y\)). We first label pinyin for incorrect characters in \(X\) and their counterparts in \(Y\) (bottom of the Figure). Then align the pinyin token according to the LCS algorithm (dashed arrows). \(X_{cor}\) is obtained by inserting pinyin tokens after the incorrect characters in \(X\). Finally, \(\hat{Y}\) is generated according to the alignment. For NULL alignments, such as “\(\vec{r}\)”, we align a special token “\(\epsilon\)”.

 

correct counterparts of \(Y\), and align according to the longest common substring (LCS) algorithm, as illustrated by the dashed arrows in the bottom of Figure 2. Then we rewrite \(Y\) to \(\hat{Y}\) according to the alignment.

The loss of the error corrector can be defined as follows:

\[\mathcal{L}_{cor}=-\sum_{i=1}^{t}\log(p(y^{\prime}_{i}=\hat{y}_{i}|X_{cor})) \tag{4}\]

Our proposed pinyin alignment provides an explicit clue for the model to learn the correlation between pinyin and Chinese characters. As an alternative, we can also learn the alignment through Connectionist Temporal Classification (CTC) Graves et al. (2006), which is widely used for _variablelength_ alignments, such as in ASR Audhkhasi et al. (2019), handwriting recognition Bluche et al. (2014), etc. But we don't use CTC in our model because we assume that aligning according to the phonological features is reliable for ASR error correction, which will reduce the learning difficulty compared with learning a global alignment. We confirm our assumption in experiments.

### Joint Learning and Inference

For each training sample \((X,Y)\), we construct its golden detection result \(C\) and correction target \(\hat{Y}\) first, then jointly optimize the two modules as follows:

\[\mathcal{L}=\lambda\cdot\mathcal{L}_{dec}+(1-\lambda)\cdot\mathcal{L}_{cor} \tag{5}\]

where \(\lambda\) is a trade-off parameter.

During the inference stage, we first detect the problematic characters and insert their pinyin into the original sentence, then predict the correction sequence \(Y^{\prime}\) by maximizing the probability of \(p(y^{\prime}_{i}|X_{cor})\) for each \(i\). The final correction result is generated by removing the blank tokens \(\epsilon\) and merging adjacent duplicate characters from the prediction of the error corrector, as shown at the top of Figure 1(d). Note that such post-processing is conducted only at locations where \(X_{cor}\) is modified relative to \(X\), while the characters predicted as correct by the error detector are kept unchanged in the final prediction.

## 3 Training Data Generation

We follow the common practice in error correction Zhang et al. (2020); Takahashi et al. (2020); Leng et al. (2021) to synthesize the training corpus with simulated ASR errors. The simulated ASR results are generated by replacing the Chinese characters or words of clean sentences into problematic ones. Specifically, we generate the noisy text in three steps: (i) sampling some candidate characters and replace them with pinyin. (ii) adding noise to the original pinyin and generate new valid ones (iii) producing new Chinese characters or words based on the updated pinyin.

In the first step, the candidate words are obtained from a confusion set Wang et al. (2019) that contains words prone to be mis-recognized. In the second step, we design three strategies of pinyin perturbation for simulating substitution errors, deletion errors and insertion errors, as shown in Figure 3. Note that when generating deletion and insertion errors, some random letters may be inserted to construct valid pinyin sequence. In the last step, we replace the noisy pinyin with corresponding word candidates, and select the sentence ranked highest by a n-gram language model as the final simulated ASR result.

Figure 3: The process of generating training samples. The characters marked in red from bottom to top represent the correct Chinese characters, the pinyin corresponding to the correct Chinese characters, the simulated pinyin disturbance, and the simulated error characters corresponding to the noisy pinyin, respectively.

 

Experiments

We carry out experiments on two Chinese ASR error correction datasets. We use word error rate (WER) and word error reduction rate (WERR) to evaluate our error correction performance.

### Data Settings

We first train an ASR model on AISHELL-1 [20]. The training set contains 150 hours of Mandarin speech, along with corresponding transcripts, mainly including news in Finance, Technology, Sports, etc. The trained ASR model then transcribes the speech to generate the paired data for evaluation, as listed in Table 1. AISHELL-1 dev and test sets contain 15 hours of speech, corresponding to 21K sentence pairs. MAGICDATA3 contains 43 hours of Mandarin speech, including interactive Q&A, daily instructions, etc.

Footnote 3: [https://openslr.org/68/](https://openslr.org/68/)

The training set of ASR error correction is constructed from 3 million web-crawled sentences, along with their noisy version with simulated errors. For the methods that only support fixed-length correction, the pseudo ASR input is generated with similar-pronounced substitution errors only. For the training data of methods that support _variable-length_ correction, we sample 2.7 million samples from the fixed-length training data and generate simulated insertion and deletion errors for the remaining 0.3M sentences.

### Training Details

Consistent with previous work [9], we use the ESPnet [17] toolkit to train an ASR model on AISHELL-1 training set. Conformer architecture [14] and SpecAugment [15] are also used to improve the ASR performance. For the ASR error correction network, we take "chinese-bert-wwm" [16] as a pre-trained model to initialize the encoders of our PhVEC and other pretraining-based methods. We set the learning rate to 5e-5 and the probability of dropout to 0.1. The loss balancing parameter \(\lambda\) in joint learning is set to 0.5, and the AdamW [12] is utilized as optimizer. More details can be found in the Appendix A.

### Baselines

We compare our PhVEC with the autoregressive Transformer [21], BART4[11] and recent state-of-the-art non-autoregressive methods as follows: **Levenshtein Transformer (LeVT)**[18] supports _variable-length_ correction by iteratively performing deletion, insertion and substitution under NAR framework. **FastCorrect**[9] implements _variable-length_ correction with a length predictor that estimates the number of target tokens each source token should be converted to, then repeating/droping the source tokens to generate a variable length sequence as the input of error correction. **MLM-phonetics**[10] implements a fixed-length error correction by pre-training a language model with phonological features integrated. **BERT** directly fine-tunes the standard masked language model to generate fixed-length corrections. **BERT+CTC** implements _variable-length_ error correction with a CTC layer built on top of BERT, which upsamples each source token twice and learns the source-target alignments automatically with a CTC loss.

Footnote 4: [https://huggingface.co/fnlp/bart-base-chinese](https://huggingface.co/fnlp/bart-base-chinese)

### Overall Results

The comparison results on two benchmark datasets are shown in Table 2. We observe that:

* Our proposed PhVEC outperforms all the other methods on the evaluation datasets. It achieves about 20% WERR for the four datasets with varying levels of ASR performance. PhVEC greatly exceeds the two existing methods supporting _variable-length_ correction, LevT, and FastCorrect, and become the first method that surpasses the autoregressive Transformer and BART models, with the WERR improved by 8.55% and 2.39% on average, respectively.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Dataset** & **AISHELL-1** & **MAGICDATA** \\ \hline Dev & 14,329 & 6,756 \\ \hline Test & 7,176 & 15,131 \\ \hline Avg Length & 14.42 & 9.78 \\ \hline Type & News & Command \\ \hline \end{tabular}
\end{table}
Table 1: Statistical results on experimental test datasets.

 

* Both PhVEC and MLM-phonetics introduce phonological features into the model but PhVEC performs better. We attribute this improvement to two aspects: one is its effective solution to _variable-length_ errors, and the other is the effective use of pinyin splitting strategy. Different from Zhang et al. (2021), we use pinyin features in letter granularity, instead of treating each pinyin as one token. This facilitates flexible insertion and enhances the correlation between similar pronounced characters and their corresponding phonological features.
* Pre-training significantly promotes correction performance. For the autoregressive models, the pre-trained BART has lower WER compared to the vanilla Transformer model. Moreover, the fine-tuned BERT achieves comparable performance with Transformer, indicating that strong language modeling will greatly facilitate NAR methods for error correction. Among the pretraining-based methods, PhVEC still performs the best, demonstrating the effectiveness of leveraging pinyin tokens for _variable-length_ correction.
* Adding the CTC layer does not bring obvious advantages for error correction. In particular, BERT+CTC is inferior to BERT, and prones to generate incorrect alignments for the fixed-length correction samples. This might be because, for ASR error correction, the alignment between most words of the source and target is definite. BERT+CTC upsamples each source token twice and dictates the model to learn the alignment, which actually increases the difficulty of learning. It is worth noting that BERT+CTC outperforms BERT on the _variable-length_ error correction samples, which will be introduced later in Table 3.
* PhVEC accelerates to 6.2 times that of Transformer, and further reduces the WER by 8-10%. Even compared with the BART model, we still have 2-3% WER reduction and speed up the inference by 7.5 times, which proves the efficiency and effectiveness of PhVEC. Moreover, we want to emphasize that the original FastCorrect used NVIDIA V100 and P40 GPU to test the model delay respectively. To ensure the consistency of the results, we reimplement all baseline methods and test the inference speed on NVIDIA V100 GPU.

It is also notable that the training data of our method PhVEC includes 3M sentence pairs, which is consistent with that of Transformer, BART, LevT, BERT, and BERT+CTC, but FastCorrect and MLM-phonetics use additional 400M and 300M pairs sentences for pre-training, respectively.

### Analyses

We further analyze the performance of our model on _variable-length_ datasets and conduct ablation

\begin{table}
\begin{tabular}{l|c c|c c|c} \hline  & \multicolumn{2}{c|}{AISHELL-1} & \multicolumn{2}{c|}{MAGICDATA} & Latency(ms/sent) \\ \cline{2-6} Method & Dev & TEST & Dev & TEST & TEST \\ \cline{2-6}  & WER\textbackslash{ (WER\(\uparrow\))} & WER\textbackslash{ (WERR\(\uparrow\))} & WER\textbackslash{ (WERR\(\downarrow\))} & WER\textbackslash{ (WER\(\uparrow\))} & GPU \\ \hline No correction & 4.46 (-) & 4.83 (-) & 13.82 (-) & 13.51 (-) & - \\ \hline \hline Transformer & 3.80 (14.80\%) & 4.08 (15.53\%) & 12.09 (12.52\%) & 12.19 (9.77\%) & 149.5 (1\(\times\)) \\ \hline BART & 3.62 (18.83\%) & 3.81 (21.12\%) & 11.36 (17.80\%) & 11.33 (16.14\%) & 180.9 (0.8\(\times\)) \\ \hline \hline LevT (Max iter=1) & 4.37 (2.02\%) & 4.73 (2.07\%) & 13.81 (0.07\%) & 13.91 (-2.96\%) & 54.0 (2.8\(\times\)) \\ \hline FastCorrect & 3.89 (12.78\%) & 4.16 (13.87\%) & - & - & 21.2 (7.1\(\times\)) \\ \hline BERT & 3.71 (16.82\%) & 3.98 (17.60\%) & 11.67 (15.56\%) & 11.79 (12.73\%) & 14.1 (10.6\(\times\)) \\ \hline BERT + CTC & 3.78 (15.25\%) & 4.01 (16.98\%) & 12.13 (12.23\%) & 12.11 (10.36\%) & 14.5 (10.3\(\times\)) \\ \hline MLM-phonetics & 3.64 (18.39\%) & 3.90 (19.25\%) & 11.59 (16.14\%) & 11.68 (13.55\%) & 23.7 (6.3\(\times\)) \\ \hline \hline PhVEC (Ours) & **3.52 (21.08\%)** & **3.62 (25.05\%)** & **11.19 (19.03\%)** & **11.04 (18.28\%)** & 24.1 (6.2\(\times\)) \\ \hline \end{tabular}
\end{table}
Table 2: Performance comparison of our method and other baselines on development set and test set. WER denotes word error rate (%), and WERR is word error reduction rate. We also test the inference speed of the correction models on NVIDIA V100 GPU and the test batch size is set to 1 sentence to match the online serving environment.

 

studies to dissect the factors affecting the effectiveness of our method.

#### 4.5.1 Variable-length scenario

To evaluate the effectiveness of our model in dealing with _variable-length_ errors, we extract the samples containing insertion and deletion errors from the test sets of AISHELL-1 and MAGICDATA, and evaluate the above methods on this subset. As shown in Table 3, PhVEC shows significant advantages over other methods, achieving over 20% WERR on both datasets. Both fined-tuned BERT and MLM-phonetics perform weakly because of the inherent fixed-length limitation of their generation. The CTC layer brings significant improvement to BERT by enabling it with _variable-length_ prediction, but there is still a large gap between its performance and that of PhVEC. The correction generated by LevT is not stable, even degrades the WERR by 8.1% WERR in the MAGICDATA subset. This might be because its length prediction result is inaccurate. The average length difference between the input and the golden correction is only 1.08, while the average length gap between the prediction of LevT and the golden correction is 1.42. This indicates that the prediction of LevT will produce more length differences than the original ASR input, resulting in its performance degradation.

#### 4.5.2 Different manners of leveraging phonological features

In our model, we split the pinyin of each Chinese character into tokens letter by letter and add them after each detected problematic character, e.g., rewrite the detected error character "[\(\ddagger\)]" (biao) to 5 tokens [\(\ddagger\)][b][i][a][o] in the intermediate _variable-length_ result. Here we explore the impact of leveraging pinyin with different strategies: 1) _OneToken_: taking the pinyin of each character as one token as in Zhang et al. (2021); 2) _Initial&Final_: divide each pinyin into an initial ([b]) and a final ([iao]), according to the phonological portion of Chinese; 3) _Letters-only_: remove the original problematic character from the rewritten sentence and use its pinyin for substitution. The first two settings focus on different pinyin granular

Figure 4: (a) and (b) are t-SNE visualization of pinyin letter and Chinese character embeddings, which correspond to BERT and PhVEC models respectively. (c) is the self-attention of the encoder layer of the error corrector. The horizontal axis represents \(X_{cor}\), the input of the error corrector, and the vertical tokens are the model output, in which [UNK] is for the blank token \(\epsilon\).

 

ities, and the third one wants to check whether the original characters detected as errors are useful for error correction.

Table 4 shows that: (1) Compared with _Letters_, both _OneToken_ and _Initial&Final_ degrades the performance. This is because the granularity of the phonological features used in the two methods are coarser than that of _Letterts_, which reduces the probability of aligning each phonological feature to more Chinese characters. (2) The _Letters-only_ performs inferior to _Letters_, demonstrating the effectiveness of keeping the original characters in \(X_{cor}\). This might be because, some wrong detection results tag "C" to some correct words. Removing these original words when adding pinyin tokens makes the model almost impossible to recover the correct ones. This always happens to rare words in some named entities.

#### 4.5.3 Characters Embedding and Alignment

To qualitatively examine whether PhVEC learns meaningful representations, we dive into the learned embedding and encoder for visualization. We first investigate whether the model learns the relationship between pinyin letters and Chinese characters. Concretely, we visualize the learned embedding of pinyin letter and Chinese character in a two-dimensional space by applying the t-SNE algorithm van der Maaten and Hinton (2008). Figure 4b shows that the embedding of Chinese characters and their corresponding pinyi letters get closer after training with PhVEC. Then we visualize the self-attention of the encoder in the error corrector. Figure 4c shows that the corrected words in the output pay much attention to their corresponding pinyin tokens, for example, "\(\top\)" is highly aligned to "b", and "\(\frac{\text{\#F}}{\text{$\times$}}\) (yao)" paid much attention to "a" and "o", which indicates that the phonological features provide an obvious prompt for error correction.

## 5 Related Work

### Chinese Spelling Error Correction (CSC)

As a close related area to ASR error correction, CSC has been widely explored in recent years. Zhang et al. (2020) proposes a soft-masked BERT model that first predicts the error probability of each character, and then uses the probabilities to perform a soft-masked word embedding for correction. As a remedy of soft-masked BERT, Zhang et al. (2021) incorporates phonological knowledge into pre-training and proposes to fuse phonological feature in error correction. Cheng et al. (2020) builds a Graph Convolution Network on top of BERT, which reflects the phonological similarity among Chinese tokens. However, these methods are designed to produce corrections of the same length as the input, but incapable of handling _variable-length_ correction that includes errors of substitution, deletion and insertion.

### Autoregressive (AR) Error Correction

To correct _variable-length_ errors, a large number of Seq2Seq AR models have been proposed. Zhang et al. (2019) uses a Transformer-based model for Chinese ASR error correction. Wang et al. (2019) incorporates a copy mechanism into Seq2Seq framework to copy the corrections directly from a prepared confusion set for the erroneous words. With the popularity of pre-training, Zhao et al. (2021) uses a pre-trained BART Lewis et al. (2020) to initialize the correction network. Although these AR models are able to deal with various types of errors in ASR, they can not satisfy the latency requirements for online services, especially for some real-time scenarios like simultaneous translation Zhang et al. (2021).

### Non-Autoregressive (NAR) Error Correction

NAR models are designed for fast generation speed compared with their AR counterpart by producing all tokens in a target sequence in parallel, which is widely explored in machine translation Gu et al. (2019); Xu and Carpuat (2021), ASR Fan et al. (2021), TTS Ren et al. (2019) etc. Recently, some studies Gu et al. (2019); Leng et al. (2021) propose to apply NAR models to _variable-length_ ASR error correction based on a length predictor, which first estimates the length of the target correction, then rewrites the input sentence by dropping or repeating some tokens according to the length estimated, finally performs a sequence labeling on the rewritten sentence to achieve correction. However, it is difficult to predict the target length of an incorrect sentence directly, even for humans, while the accuracy of length prediction is closely related to the performance of error correction.

## 6 Conclusion

In this paper, we propose a non-autoregressive Chinese ASR error correction network with phonolog 

ical training. Our method first detects the problematic characters, then adds the phonological features of them to adjust the input length, thus generating a _variable-length_ sequence for error correction. The phonological features enable our model to produce similar-pronounced corrections, and support _variable-length_ correction in a non-autoregressive mode. Experiments show that our method is superior to the autoregressive method while maintaining a 6.2x speed-up. As a future work, we plan to extend PhVEC to other languages and use corresponding phonological tokens to correct the variable length errors caused by pronunciation.

## Acknowledgements

This research is supported by the National Key Research and Development Program of China (NO\(\_\)2017YFC0820700) and National Natural Science Foundation of China (No.61902394). We thank all authors for their contributions and all anonymous reviewers for their constructive comments.

## References

* A. Anandkarni, A. Sangroya, M. Rawat, and A. Chhabra (2018)Repairing ASR output by artificial development and ontology based learning. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, pp. 5799-5801. External Links: Link Cited by: SS1.
* K. Audhkhai, G. Saon, Z. Tuske, B. Kingsbury, and M. Picheny (2019)Forget a bit to learn better: soft forgetting for ctc-based automatic speech recognition. In Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pp. 2618-2622. External Links: Link Cited by: SS1.
* Second International Conference, SLSP 2014, Lecture Notes in Computer Science, Vol. 8791, pp. 199-210. External Links: Link Cited by: SS1.
* H. Bu, J. Du, X. Na, B. Wu, and H. Zheng (2017)Aishell-1: an open-source mandarin speech corpus and a speech recognition baseline. In Oriental COCOSDA 2017, Y. Cheng, W. Xu, K. Chen, S. Jiang, F. Wang, T. Wang, W. Chu, and Y. Qi (Eds.), pp. . Note: Spellgcn: Incorporating phonological and visual similarities into language models for chinese spelling check. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, pp. 871-881. External Links: Link Cited by: SS1.
* Y. Cui, W. Che, T. Liu, B. Qin, Z. Yang, S. Wang, and G. Hu (2019)Pretraining with whole word masking for chinese BERT. CoRRabs/1906.08101. External Links: Link Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, pp. 4171-4186. External Links: Link Cited by: SS1.
* R. Fan, W. Chu, P. Chang, J. Xiao, and A. Alwan (2021)An improved single step non-autoregressive transformer for automatic speech recognition. CoRRabs/2106.09885. External Links: Link Cited by: SS1.
* A. Graves, S. Fernandez, E. J. Gomez, and J. Schmidhuber (2006)Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Machine Learning, Proceedings of the Twenty-Third International Conference (ICML 2006), Vol. 148 of ACM International Conference Proceeding Series, pp. 369-376. External Links: Link Cited by: SS1.
* J. Gu, J. Bradbury, C. Xiong, V. O. K. Li, and R. Socher (2018)Non-autoregressive neural machine translation. In 6th International Conference on Learning Representations, ICLR 2018, OpenReview.net. External Links: Link Cited by: SS1.
* J. Guo, C. Wang, and J. Zhao (2019)Levenshtein transformer. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, pp. 11179-11189. External Links: Link Cited by: SS1.
* A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang (2020)Conformer: convolution-augmented transformer for speech recognition. In Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, pp. 5036-5040. External Links: Link Cited by: SS1.
* J. Guo, T. N. Sainath, and R. J. Weiss (2019)A spelling correction model for end-to-end speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019, pp. 5651-5655. External Links: Link Cited by: SS1.
* Q. Hrinchuk, M. Popova, and B. Ginsburg (2020)Correction of automatic speech recognition with transformer sequence-to-sequence model. In 2020 IEEE International Conference on Acoustics, 

Speech and Signal Processing, ICASSP 2020_, pages 7074-7078. IEEE.
* Leng et al. (2021) Yichong Leng, Xu Tan, Linchen Zhu, Jin Xu, Renqian Luo, Linquan Liu, Tao Qin, Xiang-Yang Li, Ed Lin, and Tie-Yan Liu. 2021. Fastcorrect: Fast error correction with edit alignment for automatic speech recognition. In _35th Conference on Neural Information Processing Systems, NeurIPS 2021_.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020_, pages 7871-7880.
* Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In _7th International Conference on Learning Representations, ICLR 2019_. OpenReview.net.
* Mani et al. (2020) Anirudh Mani, Shruti Palaskar, Nimshi Venkat Meripo, Sandeep Konam, and Florian Metze. 2020. ASR error correction and domain adaptation using machine translation. In _2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020_, pages 6344-6348. IEEE.
* Park et al. (2019) Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019. Specaugment: A simple data augmentation method for automatic speech recognition. In _Interspeech 2019, 20th Annual Conference of the International Speech Communication Association_, pages 2613-2617. ISCA.
* Ren et al. (2020) Yi Ren, Jinglin Liu, Xu Tan, Zhou Zhao, Sheng Zhao, and Tie-Yan Liu. 2020. A study of non-autoregressive model for sequence generation. _arXiv preprint arXiv:2004.10454_.
* Ren et al. (2019) Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2019. Fastspeech: Fast, robust and controllable text to speech. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019_, pages 3165-3174.
* Sun et al. (2020) Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie 2.0: A continual pre-training framework for language understanding. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 34, pages 8968-8975.
* Takahashi et al. (2020) Yujin Takahashi, Sadoru Katsumata, and Mamoru Komaschi. 2020. Grammatical error correction using pseudo learner corpus considering learner's error tendency. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, ACL 2020, Online, July 5-10, 2020_, pages 27-32. Association for Computational Linguistics.
* van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. _Journal of Machine Learning Research_, 9(86):2579-2605.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017_, pages 5998-6008.
* Wang et al. (2019) Dingmin Wang, Yi Tay, and Li Zhong. 2019. Confusionset-guided pointer networks for chinese spelling check. In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019_, pages 5780-5785. Association for Computational Linguistics.
* Watanabe et al. (2018) Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Aditya Renduchintala, and Tsubasa Ochiai. 2018. Espnet: End-to-end speech processing toolkit. In _Interspeech 2018, 19th Annual Conference of the International Speech Communication Association, Hyderabad, India, 2-6 September 2018_, pages 2207-2211. ISCA.
* Xu and Carpuat (2021) Weijja Xu and Marine Carpuat. 2021. EDITOR: an edit-based transformer with repositioning for neural machine translation with soft lexical constraints. _Trans. Assoc. Comput. Linguistics_, 9:311-328.
* Zhang et al. (2021a) Ruiqing Zhang, Chao Pang, Chuanqiang Zhang, Shuohuan Wang, Zhongjun He, Yu Sun, Hua Wu, and Haifeng Wang. 2021a. Correcting chinese spelling errors with phonetic pre-training. In _Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021_, pages 2250-2261. Association for Computational Linguistics.
* Zhang et al. (2021b) Ruiqing Zhang, Xiyang Wang, Chuanqiang Zhang, Zhongjun He, Hua Wu, Zhi Li, Haifeng Wang, Ying Chen, and Qinfei Li. 2021b. BSTC: A large-scale Chinese-English speech translation dataset. In _Proceedings of the Second Workshop on Automatic Simultaneous Translation_.
* Zhang et al. (2020) Shaohua Zhang, Haoran Huang, Jicong Liu, and Hang Li. 2020. Spelling error correction with soft-masked BERT. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 882-890. Association for Computational Linguistics.
* Zhang et al. (2019) Shiliang Zhang, Ming Lei, and Zhijie Yan. 2019. Investigation of transformer based spelling correction model for ctc-based end-to-end mandarin speech recognition. In _Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019_, pages 2180-2184. ISCA 

Yun Zhao, Xuerui Yang, Jinchao Wang, Yongyu Gao, Chao Yan, and Yuanfu Zhou. 2021. BART based semantic correction for mandarin automatic speech recognition system. _CoRR_, abs/2104.05507.

## Appendix A Experimental Details

### Structure and parameters of ASR model

The ASR model is an end-to-end encoder-attention-decoder model with a 12-layer conformer encoder and a 6-layer conformer decoder, which is trained with cross-entropy loss on decoder output and an auxiliary CTC loss on encoder output. For the hyper-parameters of the ASR model, we take the beam search decoding with beam size to be 10, conformer kernel size to be 15, ctc weight to be 0.6, lm weight to be 0.3.

### Balance the objective of detection and correction

We explore the impact of the weighting strategy that balances the two objectives in fine-tuning. Table 5 presents the results of PhVEC in different values of hyper-parameter \(\lambda\). Specifically, a larger \(\lambda\) value means a higher weight on error detection, and the highest F1 score is obtained when \(\lambda\) is 0.5.