SlowGuess/NAACL_2022_OCR huggingface dataset:
It is also notable that the training data of our method PhVEC includes 3M sentence pairs, which is consistent with that of Transformer, BART, LevT, BERT, and BERT+CTC, but FastCorrect and MLM-phonetics use additional 400M and 300M pairs sentences for pre-training, respectively.
4.5 Analyses
We further analyze the performance of our model on variable-length datasets and conduct ablation
(a). BERT-tSNE
(b).PhVEC-tSNE
Figure 4: (a) and (b) are t-SNE visualization of pinyin letter and Chinese character embeddings, which correspond to BERT and PhVEC models respectively. (c) is the self-attention of the  $3^{rd}$  encoder layer of the error corrector. The horizontal axis represents  $X_{cor}$ , the input of the error corrector, and the vertical tokens are the model output, in which [UNK] is for the blank token  $\epsilon$ .
(c).PhVEC-attention
Method	AISHELL-1	MAGICDATA
No correction	15.02 (-)	21.38 (-)
Transformer	13.90 (7.5%)	20.68 (3.3%)
BART	12.08 (19.6%)	17.30 (19.1%)
LevT	14.34 (4.5%)	23.11 (-8.1%)
BERT	14.59 (2.9%)	20.08 (6.1%)
BERT+CTC	12.98 (13.6%)	19.67 (8.0%)
MLM-phonetics	14.48 (3.6%)	20.12 (5.9%)
PhVEC (Ours)	11.34 (24.5%)	16.06 (24.9%)
studies to dissect the factors affecting the effectiveness of our method.
4.5.1 Variable-length scenario
To evaluate the effectiveness of our model in dealing with variable-length errors, we extract the samples containing insertion and deletion errors from the test sets of AISHELL-1 and MAGICDATA, and evaluate the above methods on this subset. As shown in Table 3, PhVEC shows significant advantages over other methods, achieving over  $20\%$  WERR on both datasets. Both fined-tuned BERT and MLM-phonetics perform weakly because of the inherent fixed-length limitation of their generation. The CTC layer brings significant improvement to BERT by enabling it with variable-length prediction, but there is still a large gap between its performance and that of PhVEC. The correction generated by LevT is not stable, even degrades the WERR by  $8.1\%$  WERR in the MAGICDATA subset. This might be because its length prediction
Table 3: The  $\mathrm{{WER}} \downarrow  \left( {\mathrm{{WERR}} \uparrow  }\right)$  evaluated on subsets that containing only the variable-length correction samples. The number of such samples in AISHELL-1 and MAG-ICDATA is 163 and 1490,respectively.
Strategy	Example	AISHELL-1	MAGICDATA
No correction	[表]	4.83 (-)	13.51 (-)
OneToken	[表][biao]	4.08 (15.5%)	12.11 (10.4%)
Initial&Final	[表][b][iao]	3.91 (19.0%)	11.68 (13.5%)
Letters-only	[b][i][a][o]	3.85 (20.3%)	11.59 (14.2%)
Letters (Ours)	[表][b][i][a][o]	3.62 (25.1%)	11.04 (18.3%)
Table 4: The  $\mathrm{{WER}} \downarrow  \left( {\mathrm{{WERR}} \uparrow  }\right)$  of different phonological features evaluated on the AISHELL-1 and MAGICDATA test sets.
result is inaccurate. The average length difference between the input and the golden correction is only 1.08, while the average length gap between the prediction of LevT and the golden correction is 1.42. This indicates that the prediction of LevT will produce more length differences than the original ASR input, resulting in its performance degradation.
4.5.2 Different manners of leveraging phonological features
In our model, we split the pinyin of each Chinese character into tokens letter by letter and add them after each detected problematic character, e.g., rewrite the detected error character "[表]" (biao) to 5 tokens [表][b][i][a][o] in the intermediate variable-length result. Here we explore the impact of leveraging pinyin with different strategies: 1) OneToken: taking the pinyin of each character as one token as in Zhang et al. (2021a); 2) Initial&Final: divide each pinyin into an initial ([b]) and a final ([iao]), according to the phonological portion of Chinese; 3) Letters-only: remove the original problematic character from the rewritten sentence and use its pinyin for substitution. The first two settings focus on different pinyin granular
ities, and the third one wants to check whether the original characters detected as errors are useful for error correction.
Table 4 shows that: (1) Compared with Letters, both Token and Initial&Final degrades the performance. This is because the granularity of the phonological features used in the two methods are coarser than that of Letters, which reduces the probability of aligning each phonological feature to more Chinese characters. (2) The Letters-only performs inferior to Letters, demonstrating the effectiveness of keeping the original characters in  $X_{cor}$ . This might be because, some wrong detection results tag "C" to some correct words. Removing these original words when adding pinyin tokens makes the model almost impossible to recover the correct ones. This always happens to rare words in some named entities.
4.5.3 Characters Embedding and Alignment
To qualitatively examine whether PhVEC learns meaningful representations, we dive into the learned embedding and encoder for visualization. We first investigate whether the model learns the relationship between pinyin letters and Chinese characters. Concretely, we visualize the learned embedding of pinyin letter and Chinese character in a two-dimensional space by applying the t-SNE algorithm (van der Maaten and Hinton, 2008). Figure 4b shows that the embedding of Chinese characters and their corresponding pinyin letters get closer after training with PhVEC. Then we visualize the self-attention of the encoder in the error corrector. Figure 4c shows that the corrected words in the output pay much attention to their corresponding pinyin tokens, for example, "不(bu)" is highly aligned to "b", and "要(yao)" paid much attention to "a" and "o", which indicates that the phonological features provide an obvious prompt for error correction.
5.1 Chinese Spelling Error Correction (CSC)
As a close related area to ASR error correction, CSC has been widely explored in recent years. Zhang et al. (2020) proposes a soft-masked BERT model that first predicts the error probability of each character, and then uses the probabilities to perform a soft-masked word embedding for correction. As a remedy of soft-masked BERT, Zhang et al. (2021a) incorporates phonological knowl
edge into pre-training and proposes to fuse phonological feature in error correction. Cheng et al. (2020) builds a Graph Convolution Network on top of BERT, which reflects the phonological similarity among Chinese tokens. However, these methods are designed to produce corrections of the same length as the input, but incapable of handling variable-length correction that includes errors of substitution, deletion and insertion.
5.2 Autoregressive (AR) Error Correction
To correct variable-length errors, a large number of Seq2Seq AR models have been proposed. Zhang et al. (2019) uses a Transformer-based model for Chinese ASR error correction. Wang et al. (2019) incorporates a copy mechanism into Seq2Seq framework to copy the corrections directly from a prepared confusion set for the erroneous words. With the popularity of pre-training, Zhao et al. (2021) uses a pre-trained BART (Lewis et al., 2020) to initialize the correction network. Although these AR models are able to deal with various types of errors in ASR, they can not satisfy the latency requirements for online services, especially for some real-time scenarios like simultaneous translation (Zhang et al., 2021b).
5.3 Non-Autoregressive (NAR) Error Correction
NAR models are designed for fast generation speed compared with their AR counterpart by producing all tokens in a target sequence in parallel, which is widely explored in machine translation (Gu et al., 2019; Xu and Carpuat, 2021), ASR (Fan et al., 2021), TTS (Ren et al., 2019) etc. Recently, some studies (Gu et al., 2019; Leng et al., 2021) propose to apply NAR models to variable-length ASR error correction based on a length predictor, which first estimates the length of the target correction, then rewrites the input sentence by dropping or repeating some tokens according to the length estimated, finally performs a sequence labeling on the rewritten sentence to achieve correction. However, it is difficult to predict the target length of an incorrect sentence directly, even for humans, while the accuracy of length prediction is closely related to the performance of error correction.
In this paper, we propose a non-autoregressive Chinese ASR error correction network with phonolog-
ical training. Our method first detects the problematic characters, then adds the phonological features of them to adjust the input length, thus generating a variable-length sequence for error correction. The phonological features enable our model to produce similar-pronounced corrections, and support variable-length correction in a non-autoregressive mode. Experiments show that our method is superior to the autoregressive method while maintaining a 6.2x speed-up. As a future work, we plan to extend PhVEC to other languages and use corresponding phonological tokens to correct the variable length errors caused by pronunciation.
This research is supported by the National Key Research and Development Program of China (NO.2017YFC0820700) and National Natural Science Foundation of China (No.61902394). We thank all authors for their contributions and all anonymous reviewers for their constructive comments.
C. Anantaram, Amit Sangroya, Mrinal Rawat, and Aishwarya Chhabra. 2018. Repairing ASR output by artificial development and ontology based learning. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, pages 5799-5801. ijcai.org.
Kartik Audhkhasi, George Saon, Zoltán Túske, Brian Kingsbury, and Michael Picheny. 2019. Forget a bit to learn better: Soft forgetting for ctc-based automatic speech recognition. In *Interspeech* 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pages 2618-2622. ISCA.
Theodore Bluche, Hermann Ney, and Christopher Kermorvant. 2014. A comparison of sequence-trained deep neural networks and recurrent neural networks optical modeling for handwriting recognition. In Statistical Language and Speech Processing - Second International Conference, SLSP 2014, volume 8791 of Lecture Notes in Computer Science, pages 199-210. Springer.
Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. 2017. Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline. In Oriental COCOSDA 2017, page Submitted.
Xingyi Cheng, Weidi Xu, Kunlong Chen, Shaohua Jiang, Feng Wang, Taifeng Wang, Wei Chu, and Yuan Qi. 2020. Spellgcn: Incorporating phonological and visual similarities into language models for chinese
spelling check. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, pages 871-881. Association for Computational Linguistics.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. Pretraining with whole word masking for chinese BERT. CoRR, abs/1906.08101.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, pages 4171-4186. Association for Computational Linguistics.
Ruchao Fan, Wei Chu, Peng Chang, Jing Xiao, and Abeer Alwan. 2021. An improved single step non-autoregressive transformer for automatic speech recognition. CoRR, abs/2106.09885.
Alex Graves, Santiago Fernández, Faustino J. Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Machine Learning, Proceedings of the Twenty-Third International Conference (ICML 2006), volume 148 of ACM International Conference Proceeding Series, pages 369-376. ACM.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In 6th International Conference on Learning Representations, ICLR 2018. OpenReview.net.
Jiatao Gu, Changhan Wang, and Junbo Zhao. 2019. Levenshtein transformer. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, pages 11179-11189.
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020. Conformer: Convolution-augmented transformer for speech recognition. In *Interspeech* 2020, 21st Annual Conference of the International Speech Communication Association, pages 5036-5040. ISCA.
Jinxi Guo, Tara N. Sainath, and Ron J. Weiss. 2019. A spelling correction model for end-to-end speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019, pages 5651-5655. IEEE.
Oleksii Hrinchuk, Mariya Popova, and Boris Ginsburg. 2020. Correction of automatic speech recognition with transformer sequence-to-sequence model. In 2020 IEEE International Conference on Acoustics,
Speech and Signal Processing, ICASSP 2020, pages 7074-7078. IEEE.
Yichong Leng, Xu Tan, Linchen Zhu, Jin Xu, Renqian Luo, Linquan Liu, Tao Qin, Xiang-Yang Li, Ed Lin, and Tie-Yan Liu. 2021. Fastcorrect: Fast error correction with edit alignment for automatic speech recognition. In 35th Conference on Neural Information Processing Systems, NeurIPS 2021.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, pages 7871-7880.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019. OpenReview.net.
Anirudh Mani, Shruti Palaskar, Nimshi Venkat Meripo, Sandeep Konam, and Florian Metze. 2020. ASR error correction and domain adaptation using machine translation. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020, pages 6344-6348. IEEE.
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019. Specaugment: A simple data augmentation method for automatic speech recognition. In *Interspeech* 2019, 20th Annual Conference of the International Speech Communication Association, pages 2613-2617. ISCA.
Yi Ren, Jinglin Liu, Xu Tan, Zhou Zhao, Sheng Zhao, and Tie-Yan Liu. 2020. A study of non-autoregressive model for sequence generation. arXiv preprint arXiv:2004.10454.
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2019. Fastspeech: Fast, robust and controllable text to speech. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, pages 3165-3174.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie 2.0: A continual pre-training framework for language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8968-8975.
Yujin Takahashi, Satoru Katsumata, and Mamoru Komachi. 2020. Grammatical error correction using pseudo learner corpus considering learner's error tendency. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, ACL 2020, Online, July 5-10, 2020, pages 27-32. Association for Computational Linguistics.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579-2605.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998-6008.
Dingmin Wang, Yi Tay, and Li Zhong. 2019. Confusionset-guided pointer networks for chinese spelling check. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, pages 5780-5785. Association for Computational Linguistics.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. 2018. Espnet: End-to-end speech processing toolkit. In *Interspeech* 2018, 19th Annual Conference of the International Speech Communication Association, Hyderabad, India, 2-6 September 2018, pages 2207-2211. ISCA.
Weijia Xu and Marine Carpuat. 2021. EDITOR: an edit-based transformer with repositioning for neural machine translation with soft lexical constraints. Trans. Assoc. Comput. Linguistics, 9:311-328.
Ruiqing Zhang, Chao Pang, Chuanqiang Zhang, Shuohuan Wang, Zhongjun He, Yu Sun, Hua Wu, and Haifeng Wang. 2021a. Correcting chinese spelling errors with phonetic pre-training. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, pages 2250-2261. Association for Computational Linguistics.
Ruiqing Zhang, Xiyang Wang, Chuanqiang Zhang, Zhongjun He, Hua Wu, Zhi Li, Haifeng Wang, Ying Chen, and Qinfei Li. 2021b. BSTC: A large-scale Chinese-English speech translation dataset. In Proceedings of the Second Workshop on Automatic Simultaneous Translation.
Shaohua Zhang, Haoran Huang, Jicong Liu, and Hang Li. 2020. Spelling error correction with soft-masked BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 882-890. Association for Computational Linguistics.
Shiliang Zhang, Ming Lei, and Zhijie Yan. 2019. Investigation of transformer based spelling correction model for ctc-based end-to-end mandarin speech recognition. In *Interspeech* 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pages 2180-2184. ISCA.
Yun Zhao, Xuerui Yang, Jinchao Wang, Yongyu Gao, Chao Yan, and Yuanfu Zhou. 2021. BART based semantic correction for mandarin automatic speech recognition system. CoRR, abs/2104.05507.
A.1 Structure and parameters of ASR model
The ASR model is an end-to-end encoder-attention-decoder model with a 12-layer conformer encoder and a 6-layer conformer decoder, which is trained with cross-entropy loss on decoder output and an auxiliary CTC loss on encoder output. For the hyper-parameters of the ASR model, we take the beam search decoding with beam size to be 10, conformer kernel size to be 15, ctc weight to be 0.6, lm weight to be 0.3.
A.2 Balance the objective of detection and correction
We explore the impact of the weighting strategy that balances the two objectives in fine-tuning. Table 5 presents the results of PhVEC in different values of hyper-parameter  $\lambda$ . Specifically, a larger  $\lambda$  value means a higher weight on error detection, and the highest F1 score is obtained when  $\lambda$  is 0.5.
λ	AISHELL-1	MAGICDATA
Dev(WER↓)	TEST(WER↓)	Dev(WER↓)	TEST(WER↓)
0.2	3.61	3.75	11.42	11.37
0.5	3.52	3.62	11.19	11.04
0.8	3.74	3.83	11.55	11.47
Table 5: Impact of Different Values of  $\lambda$  .