SlowGuess/NAACL_2022_OCR huggingface dataset:
Zheng Fang $^{1,2}$ , Ruiqing Zhang $^{3*}$ , Zhongjun He $^{3}$ , Hua Wu $^{3}$ , Yanan Cao $^{1,2}$
$^{1}$ Institute of Information Engineering, Chinese Academy of Sciences
$^{2}$ School of Cyber Security, University of Chinese Academy of Sciences
<sup>3</sup>Baidu Inc. No. 10, Shangdi 10th Street, Beijing, 100085, China
1,2{fangzheng，caoyanan}@iie.ac.cn
3{zhangruiqing01, hezhongjun, wu_hua}@baidu.com
Automatic Speech Recognition (ASR) is an efficient and widely used input method that transcribes speech signals into text. As the errors introduced by ASR systems will impair the performance of downstream tasks, we introduce a post-processing error correction method, PhVEC, to correct errors in text space. For the errors in ASR result, existing works mainly focus on fixed-length corrections, modifying each wrong token to a correct one (one-to-one correction), but rarely consider the variable-length correction (one-to-many or many-to-one correction). In this paper, we propose an efficient non-autoregressive (NAR) method for Chinese ASR error correction for both cases. Instead of conventionally predicting the sentence length in NAR methods, we propose a novel approach that uses phonological tokens to extend the source sentence for variable-length correction, enabling our model to generate phonetically similar corrections. Experimental results on datasets of different domains show that our method achieves significant improvement in word error rate reduction and speeds up the inference by 6.2 times compared with the autoregressive model.
Errors introduced by automatic speech recognition (ASR) usually affect the performance of downstream tasks such as phonetic search, speech translation, etc. In recent years, ASR error correction techniques have been proposed (Anantaram et al., 2018; Mani et al., 2020; Zhao et al., 2021; Leng et al., 2021) to refine the ASR output and correct errors in text space. Without loss of generality, we study Chinese ASR error correction in this paper.
Given the ASR result of an utterance, the goal of error correction is to generate a sentence with the wrongly recognized words corrected. Thus the
ASR error correction can be modeled as a machine translation problem under conventional autoregressive sequence-to-sequence (Seq2Seq) framework (Guo et al., 2019; Hrinchuk et al., 2020; Mani et al., 2020). However, the autoregressive models suffer from inefficient decoding since the generation of each target token depends on previously generated characters (Figure 1 (a)). Furthermore, without considering the phonetic similarities, the method is prone to generate corrections with totally different pronunciation, as for the example, the error character (“表”, biao) should be corrected into (“不要”, bu yao) but the Seq2Seq model ignores phonological features and corrects it to a phonetically different correction (“不许”, bu xu).
To speed up prediction, recent studies propose to take non-autoregressive (NAR) methods (Gu et al., 2018; Ren et al., 2020) for error correction, which generates target tokens in parallel. Most NAR approaches make fixed-length predictions that generate same-length output as the source input by directly tagging on the source text (Zhang et al., 2020). Some works further leverage phonological features to correct the ASR errors caused by similar pronounced characters (Zhang et al., 2021a; Cheng et al., 2020). Such methods have made great improvements in correcting simple errors, but cannot handle samples with different lengths of source and target, referred to as variable-length error correction. See Figure 1 (b) for illustration. The method successfully finds out the erroneous character (“表”, biao), and substitutes it with its phonological feature “biao” for error correction. However, it eventually generates a wrong correction (“标”, biao) under the constraint of fixed length prediction.
To overcome above constraint while preserving efficient prediction, NAR solutions for variable-length prediction are proposed (Leng et al., 2021; Gu et al., 2019). They first build a length predictor to estimate fertility, i.e., the number of target tokens
* This work was conducted at Baidu. Corresponding author: Ruiqing Zhang.
Human Language Technologies, pages 5907 - 5917
(a). Seq2Seq
(b). Tagging with Phonetics
(c). NAR with Length Predictor
Figure 1: An example of Chinese ASR Error Correction. The source speech "ni bu yao nan guo" ("Don't be sad") is incorrectly recognized to "ni biao nan guo" ("Your watch is sad"). (a) The Seq2Seq model generates a fluent but incorrect sentence because it neglects the phonetic similarity. (b) The sequence tagging model with phonological features correctly detects the character with error ("K" means keep unchanged and "C" means to be correct), but because it only supports equal-length prediction, it fails to generate a fluent correction with equal length as the input. (c) The NAR model with a length predictor incorrectly predicts the number of target tokens corresponding to the source character "ni" as 0, which should be 1. Accordingly, it removes the character and generates an incorrect result. (d) The proposed PhVEC first detects problematic tokens, then leverages their phonological features to generate a variable-length intermediate result, and outputs the final result by reducing the repeated characters.
(d). Phonology based NAR (Our proposed)
aligned to each source token, then up-sampling or dropping some source tokens accordingly for parallel correction. However, the method is fragile because incorrect length prediction will distort the meaning of the sentence and may directly lead to false prediction. See Figure 1 (c) for example, the length predictor produces a wrong alignment (0 2 1 1), indicating that the decoder will drop the first source character and generate two target characters for the second source character. According to the alignment, the intermediate result turns to be a meaningless sentence ("表表难过", means "watch watch sad") and leads to a wrong correction.
To address the issues mentioned above, we propose a novel NAR model named PhVEC (Phonology-based Variable-length Error Correction). The model incorporates the phonological features of error characters to enable variable-length prediction. Concretely, PhVEC contains a detection network and a correction network based on a pre-trained language model (Devlin et al., 2019; Sun et al., 2020). The detection network predicts the correctness of each token, and the correction network generates correction result. Instead of using a length predictor to predict the fertility for each source token, we insert the phonological tokens (Chinese Pinyin<sup>1</sup>) to the source sequence as placeholders after each detected erroneous character. During prediction, each source token (either
the character token or pinyin token) can generate zero or one target token. With the guidance of the phonological token, the model will generate characters with similar pronunciations. As Figure 1 (d) shows, the pinyin token “ $b$ ” produces target character (“不”,  $bu$  ) with the same consonant letter, and “ $a o$ ” produces target character (“要”,  $yao$  ) with the same vowel letter. We delete the repeated characters in the final sequence.
We evaluate our methods on multiple datasets with varying degrees of ASR word error rates. Experimental results show that our PhVEC obtains significant improvement (over  $10\%$  word error rate reduction) on standard benchmarks compared with existing NAR-based ASR error correction methods at comparable speed. Even compared with the AR baseline Transformer and BART (Lewis et al., 2020) models, PhVEC can still have an  $8.55\%$  and  $2.39\%$  word error rate reduction while keeping a  $6.2\mathrm{x}$  speed-up.
Chinese ASR error correction can be formalized as the following task. Given a speech recognition sequence  $X = (x_{1}, x_{2}, \ldots, x_{n})$  of  $n$  Chinese characters, the goal is to correct it into another sequence of  $m$  characters  $Y = (y_{1}, y_{2}, \ldots, y_{m})$ . Note that the target sequence length  $m$  does not have to be equal to the source sequence length  $n$ . There exist three types of ASR errors in transforming from  $X$  to  $Y$ : substitution, deletion, and insertion.
As illustrated in Figure 1(d), our proposed ASR
<sup>1</sup>In this paper, we ignore the tone of pinyin, and the pinyin of each Chinese character can be represented by one to six English letters.
error correction model is composed of an error detector and an error corrector, both perform a NAR tagging. The error detector takes  $X$  as input to predict the correctness of each token. The error corrector takes the combination of the source tokens and the phonological features of problematic tokens as input for correction. During error correction, each token of the extended input can be tagged to a Chinese character or a blank token “ $\epsilon$ ”, so that the model can support variable-length correction. Since different pinyin letters may generate the same character, we eliminate continuous repetitive characters in the final result. In this paper, we use Chinese pinyin as the phonological feature. The learning of PhVEC is conducted end-to-end, with the error detector and corrector optimized jointly.
2.1 Error Detection Network
The goal of the error detector is to check whether a character  $x_{i}$  ( $1 \leq i \leq n$ ) is correct or not. We model this task as sequence labeling. We build a sequential binary classifier and use class 1 and 0 to label the problematic characters and correct characters, respectively. The ground-truth detection label  $C = (c_{1}, c_{2}, \ldots, c_{n})$  is pre-calculated by matching  $X$  and  $Y$  with edit distance, in which  $c_{i} \in \{0,1\}$ . The prediction result of the error detector is represented by a sequence  $C' = (c_{1}', c_{2}', \ldots, c_{n}')$ , and we use  $p_{i}$  to denote the probability of token  $x_{i}$  being predicted to class 1, which can be formalized as follows:
$$
p _ {i} = p \left(c _ {i} ^ {\prime} = 1 | X\right) = \operatorname {s o f t m a x} \left(f _ {\text {d e c}} \left(E \left(\boldsymbol {e} _ {\boldsymbol {w}}\right)\right)\right) \tag {1}
$$
where  $e_{\mathbf{w}} = (e_{x_1}, e_{x_2}, \dots, e_{x_n})$  is the token embedding of  $X$ ,  $E$  is a Transformer-based encoder and  $f_{dec}$  is a fully-connected layer that maps the sentence representation to a binary sequence. To train the model, we adopt the following cross entropy loss function:
$$
\mathcal {L} _ {d e c} = - \frac {1}{n} \sum_ {i} \left[ c _ {i} \ln p _ {i} + \left(1 - c _ {i}\right) \ln \left(1 - p _ {i}\right) \right] \tag {2}
$$
2.2 Error Correction Network
After identifying the errors, we introduce the pinyin features of the incorrect characters and feed them into the correction network. Concretely, we use the tool PyPinyin<sup>2</sup> to generate pinyin for each erroneous Chinese character and insert pinyin tokens after the problematic token. The original
Figure 2: An illustration of generating  $X_{cor}$  and  $\hat{Y}$  from the training sample  $(X, Y)$ . We first label pinyin for incorrect characters in  $X$  and their counterparts in  $Y$  (bottom of the Figure). Then align the pinyin token according to the LCS algorithm (dashed arrows).  $X_{cor}$  is obtained by inserting pinyin tokens after the incorrect characters in  $X$ . Finally,  $\hat{Y}$  is generated according to the alignment. For NULL alignments, such as "i", we align a special token "ε".
sentence  $X = (x_{1},x_{2},\ldots ,x_{n})$  is thus rewritten as  $X_{cor} = (x_1',x_2',\dots,x_t')$  , where  $t\geq n$  , and  $t - n$  is the number of pinyin tokens.
Given  $X_{cor}$ , the error corrector performs sequence labeling to predict the correction result  $Y' = (y_1', y_2', \dots, y_t')$  as follows:
$$
p \left(y _ {i} ^ {\prime} = \mathcal {V} _ {j} \mid X _ {c o r}\right) = \operatorname {s o f t m a x} \left(f _ {c r t} E \left(\boldsymbol {e} _ {\boldsymbol {c o r}}\right)\right) \tag {3}
$$
where  $p(y_i' = \mathcal{V}_j | X_{cor})$  is the conditional probability that  $x_i'$  is corrected to  $\mathcal{V}_j$ , the token with index  $j$  in the vocabulary  $\mathcal{V}$ .  $e_{cor}$  is the token embedding of  $X_{cor}$  and  $f_{crt}$  is a fully-connected layer that maps the hidden states of source tokens to their predicted logit vector of length  $|\mathcal{V}|$ . Note that, the parameters of the token embedding, the encoder  $E$  and the correction network  $f_{crt}$  are initialized by pre-trained language models (Devlin et al., 2019; Sun et al., 2020). We share the pre-training model parameters in the error detector and corrector to encode the Chinese characters and pinyin tokens in a shared space, which not only reduces the model size, but also makes the model yield better semantic representation.
The learning objective of the error corrector is to correct  $X_{cor}$  to the golden correction  $Y$ . However, it is not an easy task because the lengths of  $X_{cor}$  and  $Y$  may not be the same. Therefore, we rewrite  $Y$  to  $\hat{Y} = (\hat{y}_1, \hat{y}_2, \dots, \hat{y}_t)$  which has the same length as  $X_{cor}$ . Concretely,  $\hat{Y}$  is constructed from  $Y$  with some tokens repeated and inserted according to pronunciation alignment. We compare the pinyin of the incorrect tokens of  $X$  with their
$^{2}$ https://github.com/mozillazg/python-pinyin
Figure 3: The process of generating training samples. The characters marked in red from bottom to top represent the correct Chinese characters, the pinyin corresponding to the correct Chinese characters, the simulated pinyin disturbance, and the simulated error characters corresponding to the noisy pinyin, respectively.
correct counterparts of  $Y$ , and align according to the longest common substring (LCS) algorithm, as illustrated by the dashed arrows in the bottom of Figure 2. Then we rewrite  $Y$  to  $\hat{Y}$  according to the alignment.
The loss of the error corrector can be defined as follows:
$$
\mathcal {L} _ {c o r} = - \sum_ {i = 1} ^ {t} \log \left(p \left(y _ {i} ^ {\prime} = \hat {y} _ {i} \mid X _ {c o r}\right)\right) \tag {4}
$$
Our proposed pinyin alignment provides an explicit clue for the model to learn the correlation between pinyin and Chinese characters. As an alternative, we can also learn the alignment through Connectionist Temporal Classification (CTC) (Graves et al., 2006), which is widely used for variable-length alignments, such as in ASR (Audhkhasi et al., 2019), handwriting recognition (Bluche et al., 2014), etc. But we don't use CTC in our model because we assume that aligning according to the phonological features is reliable for ASR error correction, which will reduce the learning difficulty compared with learning a global alignment. We confirm our assumption in experiments.
2.3 Joint Learning and Inference
For each training sample  $(X,Y)$ , we construct its golden detection result  $C$  and correction target  $\hat{Y}$  first, then jointly optimize the two modules as follows:
$$
\mathcal {L} = \lambda \cdot \mathcal {L} _ {d e c} + (1 - \lambda) \cdot \mathcal {L} _ {c o r} \tag {5}
$$
During the inference stage, we first detect the problematic characters and insert their pinyin into the original sentence, then predict the correction sequence  $Y'$  by maximizing the probability of  $p(y_i'|X_{cor})$  for each  $i$ . The final correction result
is generated by removing the blank tokens  $\epsilon$  and merging adjacent duplicate characters from the prediction of the error corrector, as shown at the top of Figure 1(d). Note that such post-processing is conducted only at locations where  $X_{cor}$  is modified relative to  $X$ , while the characters predicted as correct by the error detector are kept unchanged in the final prediction.
3 Training Data Generation
We follow the common practice in error correction (Zhang et al., 2020; Takahashi et al., 2020; Leng et al., 2021) to synthesize the training corpus with simulated ASR errors. The simulated ASR results are generated by replacing the Chinese characters or words of clean sentences into problematic ones. Specifically, we generate the noisy text in three steps: (i) sampling some candidate characters and replace them with pinyin. (ii) adding noise to the original pinyin and generate new valid ones (iii) producing new Chinese characters or words based on the updated pinyin.
In the first step, the candidate words are obtained from a confusion set (Wang et al., 2019) that contains words prone to be mis-recognized. In the second step, we design three strategies of pinyin perturbation for simulating substitution errors, deletion errors and insertion errors, as shown in Figure 3. Note that when generating deletion and insertion errors, some random letters may be inserted to construct valid pinyin sequence. In the last step, we replace the noisy pinyin with corresponding word candidates, and select the sentence ranked highest by a n-gram language model as the final simulated ASR result.
We carry out experiments on two Chinese ASR error correction datasets. We use word error rate (WER) and word error reduction rate (WERR) to evaluate our error correction performance.
4.1 Data Settings
We first train an ASR model on AISHELL-1 (Bu et al., 2017). The training set contains 150 hours of Mandarin speech, along with corresponding transcripts, mainly including news in Finance, Technology, Sports, etc. The trained ASR model then transcribes the speech to generate the paired data for evaluation, as listed in Table 1. AISHELL-1 dev and test sets contain 15 hours of speech, corresponding to 21K sentence pairs. MAGICDATA<sup>3</sup> contains 43 hours of Mandarin speech, including interactive Q&A, daily instructions, etc.
The training set of ASR error correction is constructed from 3 million web-crawled sentences, along with their noisy version with simulated errors. For the methods that only support fixed-length correction, the pseudo ASR input is generated with similar-pronounced substitution errors only. For the training data of methods that support variable-length correction, we sample 2.7 million samples from the fixed-length training data and generate simulated insertion and deletion errors for the remaining 0.3M sentences.
Consistent with previous work (Leng et al., 2021), we use the ESPnet (Watanabe et al., 2018) toolkit to train an ASR model on AISHELL-1 training set. Conformer architecture (Gulati et al., 2020) and SpecAugment (Park et al., 2019) are also used to improve the ASR performance. For the ASR error correction network, we take "chinese-bert-wwm" (Cui et al., 2019) as a pre-trained model to initialize the encoders of our PhVEC and other pretraining-based methods. We set the learning rate to 5e-5 and the probability of dropout to 0.1. The loss balancing parameter  $\lambda$  in joint learning is set to 0.5, and the AdamW (Loshchilov and Hutter, 2019) is utilized as optimizer. More details can be found in the Appendix A.
Dataset	AISHELL-1	MAGICDATA
Dev	14,329	6,756
Test	7,176	15,131
Avg Length	14.42	9.78
Type	News	Command
Table 1: Statistical results on experimental test datasets.
We compare our PhVEC with the autoregressive Transformer (Vaswani et al., 2017), BART $^4$  (Lewis et al., 2020) and recent state-of-the-art non-autoregressive methods as follows: Levenshtein Transformer (LevT) (Gu et al., 2019) supports variable-length correction by iteratively performing deletion, insertion and substitution under NAR framework. FastCorrect (Leng et al., 2021) implements variable-length correction with a length predictor that estimates the number of target tokens each source token should be converted to, then repeating/dropping the source tokens to generate a variable length sequence as the input of error correction. MLM-phonetics (Zhang et al., 2021a) implements a fixed-length error correction by pre-training a language model with phonological features integrated. BERT directly fine-tunes the standard masked language model to generate fixed-length corrections. BERT+CTC implements variable-length error correction with a CTC layer built on top of BERT, which upsamples each source token twice and learns the source-target alignments automatically with a CTC loss.
The comparison results on two benchmark datasets are shown in Table 2. We observe that:
- Our proposed PhVEC outperforms all the other methods on the evaluation datasets. It achieves about  $20\%$  WERR for the four datasets with varying levels of ASR performance. PhVEC greatly exceeds the two existing methods supporting variable-length correction, LevT, and FastCorrect, and become the first method that surpasses the autoregressive Transformer and BART models, with the WERR improved by  $8.55\%$  and  $2.39\%$  on average, respectively.
3https://openslr.org/68/
<sup>4</sup>https://huggingface.co/fnlp/bart-base-chinese
Method	AISHELL-1	MAGICDATA	Latency(ms/sent)
Dev	TEST	Dev	TEST	TEST
WER↓ (WERR↑)	WER↓ (WERR↑)	WER↓ (WERR↑)	WER↓ (WERR↑)	GPU
No correction	4.46 (-)	4.83 (-)	13.82 (-)	13.51 (-)	-
Transformer	3.80 (14.80%)	4.08 (15.53%)	12.09 (12.52%)	12.19 (9.77%)	149.5 (1×)
BART	3.62 (18.83%)	3.81 (21.12%)	11.36 (17.80%)	11.33 (16.14%)	180.9 (0.8×)
LevT (Max iter=1)	4.37 (2.02%)	4.73 (2.07%)	13.81 (0.07%)	13.91 (-2.96%)	54.0 (2.8×)
FastCorrect	3.89 (12.78%)	4.16 (13.87%)	-	-	21.2 (7.1×)
BERT	3.71 (16.82%)	3.98 (17.60%)	11.67 (15.56%)	11.79 (12.73%)	14.1 (10.6×)
BERT + CTC	3.78 (15.25%)	4.01 (16.98%)	12.13 (12.23%)	12.11 (10.36%)	14.5 (10.3×)
MLM-phonetics	3.64 (18.39%)	3.90 (19.25%)	11.59 (16.14%)	11.68 (13.55%)	23.7 (6.3×)
PhVEC (Ours)	3.52 (21.08%)	3.62 (25.05%)	11.19 (19.03%)	11.04 (18.28%)	24.1 (6.2×)
Table 2: Performance comparison of our method and other baselines on development set and test set. WER denotes word error rate (\%), and WERR is word error reduction rate. We also test the inference speed of the correction models on NVIDIA V100 GPU and the test batch size is set to 1 sentence to match the online serving environment.
- Both PhVEC and MLM-phonetics introduce phonological features into the model but PhVEC performs better. We attribute this improvement to two aspects: one is its effective solution to variable-length errors, and the other is the effective use of pinyin splitting strategy. Different from Zhang et al. (2021a), we use pinyin features in letter granularity, instead of treating each pinyin as one token. This facilitates flexible insertion and enhances the correlation between similar pronounced characters and their corresponding phonological features.
- Pre-training significantly promotes correction performance. For the autoregressive models, the pre-trained BART has lower WER compared to the vanilla Transformer model. Moreover, the fine-tuned BERT achieves comparable performance with Transformer, indicating that strong language modeling will greatly facilitate NAR methods for error correction. Among the pretraining-based methods, PhVEC still performs the best, demonstrating the effectiveness of leveraging pinyin tokens for variable-length correction.
- Adding the CTC layer does not bring obvious advantages for error correction. In particular, BERT+CTC is inferior to BERT, and prones to generate incorrect alignments for the fixed-length correction samples. This might be because, for ASR error correction, the alignment
between most words of the source and target is definite. BERT+CTC upsamples each source token twice and dictates the model to learn the alignment, which actually increases the difficulty of learning. It is worth noting that BERT+ CTC outperforms BERT on the variable-length error correction samples, which will be introduced later in Table 3.
- PhVEC accelerates to 6.2 times that of Transformer, and further reduces the WER by  $8 - 10\%$ . Even compared with the BART model, we still have  $2 - 3\%$  WER reduction and speed up the inference by 7.5 times, which proves the efficiency and effectiveness of PhVEC. Moreover, we want to emphasize that the original FastCorrect used NVIDIA V100 and P40 GPU to test the model delay respectively. To ensure the consistency of the results, we reimplement all baseline methods and test the inference speed on NVIDIA V100 GPU.