SlowGuess/NAACL_2021_OCR huggingface dataset:
Yosuke Higuchi, Shinji Watanabe, Nanxin Chen, Tetsuji Ogawa, and Tetsunori Kobayashi. 2020. Mask CTC: Non-autoregressive end-to-end ASR with CTC and mask predict. In INTERSPEECH. ISCA.
Hirofumi Inaguma, Yosuke Higuchi, Kevin Duh, Tatsuya Kawahara, and Shinji Watanabe. 2021. Orthros: Non-autoregressive end-to-end speech translation with dual-decoder. In ICASSP. IEEE.
Shigeki Karita, Xiaofei Wang, Shinji Watanabe, Takenori Yoshimura, Wangyou Zhang, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, and Ryuichi Yamamoto. 2019. A comparative study on transformer vs RNN in speech applications. In ASRU, pages 449-456. IEEE.
Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah A. Smith. 2021. Deep encoder, shallow decoder: Reevaluating the speed-quality tradeoff in machine translation. In ICLR.
Samuel Kriman, Stanislav Beliaev, Boris Ginsburg, Jocelyn Huang, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, and Yang Zhang. 2020. QuartzNet: Deep automatic speech recognition with 1d time-channel separable convolutions. In ICASSP, pages 6124-6128. IEEE.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In EMNLP, pages 1173-1182. Association for Computational Linguistics.
Toan Q. Nguyen and Julian Salazar. 2019. Transformers without tears: Improving the normalization of self-attention. In IWSLT.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. LibriSpeech: An ASR corpus based on public domain audio books. In ICASSP, pages 5206-5210. IEEE.
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019. SpecAugment: A simple data augmentation method for automatic speech recognition. In INTERSPEECH, pages 2613-2617. ISCA.
Douglas B. Paul and Janet M. Baker. 1992. The design for the Wall Street Journal-based CSR corpus. In ICSLP. ISCA.
Julian Salazar, Katrin Kirchhoff, and Zhiheng Huang. 2019. Self-attention networks for connectionist temporal classification in speech recognition. In ICASSP, pages 7115-7119. IEEE.
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Kartrin Kirchhoff. 2020. Masked language model scoring. In ACL, pages 2699-2712. Association for Computational Linguistics.
Zhengkun Tian, Jiangyan Yi, Ye Bai, Jianhua Tao, Shuai Zhang, and Zhengqi Wen. 2021. One in a hundred: Select the best predicted sequence from numerous candidates for streaming speech recognition. CoRR, abs/2010.14791v3.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS, pages 5998-6008.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. 2018. ESPnet: End-to-end speech processing toolkit. In INTERSPEECH, pages 2207-2211. ISCA.
A Additional Analysis
Character error rates (CERs). Given that the WSJ models operate on characters, we also report model CERs (where available) in Table 4, as a direct counterpart to Table 1:
Model	# passes Enc/Dec	dev93	eval92
CER	WER	CER	WER
Autoregressive baseline (Higuchi et al., 2020)
CTC+ATTENTION‡	1/L	5.5	14.4	4.0	11.3
+ beam search	1/>L	-	13.5	-	10.9
Previous work (Higuchi et al., 2020; Chan et al., 2020)
CTC	1/-	-	22.2	-	17.9
MASK-CTC	1/0	-	16.3	-	12.9
1/1	-	15.7	-	12.5
1/5	-	15.5	-	12.2
1/#msk.	-	15.4	-	12.1
IMPUTER (8-LYR) †	8/-	-	-	4.9†	12.7
Our work
CTC	1/-	5.24	18.6	4.16	15.0
MASK-CTC	1/5	4.73	15.3	3.88	12.8
ALIGN-REFINE	1/0	4.75	16.2	3.92	13.5
1/1	4.38	14.1	3.61	11.6
1/3	4.34	13.9	3.61	11.5
1/5	4.33	13.7	3.60	11.4
1/10	4.33	13.7	3.60	11.4
Table 4: Non-autoregressive ASR on WSJ. No LMs are used. †: No SpecAugment, uses 400 BPE vocabulary. ‡: CERs retrieved from https://github.com/espnet/espnet/blob/ffc39c27f9aa0cf15c7ae7a06a8a9d35871602e/egs/wsj/asr1/RESULTSLS.md#cer-4.
In Align-Refine, we see that an absolute reduction in CER per realignment leads to a 5x to 20x absolute reduction in WER, suggesting the corrected errors are largely character-level misspellings. Align-Refine also has smaller CERs for the comparable WER to the CTC+attention autoregressive model, e.g., Align-Refine has a CER of
4.4 for a WER of 14.1 while CTC+Attention has a CER of 5.5 for a WER of 14.4. This exhibits one difference in failure modes: when an autoregressive model predicts the wrong word, it is more likely to get the entire word wrong due to compounding error.
Speaking rate sensitivity. For some insight into how Align-Refine performance varies with approximate rate of speech, we split LibriSpeech test by the ratio of reference words to alignment length (4 × downsampled from regular audio frames). Our results are in Table 5:
words / align. lbl.	test-clean	test-other
words/utt.	WER	words/utt.	WER
[0.02, 0.09)	13.4	5.66%	10.1	14.10%
[0.09, 0.11)	21.2	3.60%	18.3	10.15%
[0.11, 0.13)	23.1	3.21%	20.9	7.58%
[0.13, 0.21)	20.0	2.99%	21.3	6.53%
Table 5: WER bucketed by approximate "speed" (as measured by words vs. alignment lengths) of both the test-clean and test-other evaluation sets. Each (bucket, set) pair contains 487 to 917 utterances.
Surprisingly, we find "slower" segments do worse (though this is partly confounded by lowword utterances having more silences). One possible explanation is that correcting a word requires coordinated change across multiple alignment labels, which is easier when there are fewer alignment labels per word. However, such "coordination" is hard as each position's edit is conditionally independent from other edits at any given iteration.
Our setup extends the Mask-CTC recipe (https: //github.com/espnet/espnet/pull/2223/) in ESPnet (Watanabe et al., 2018). We release our AlignRefine recipe and ESPnet code changes at https: //github.com/amazon-research/align-refine.
Architecture. Following Mask-CTC, the 12-layer encoder has a convolutional frontend that downsamples input lengths and features by  $4 \times$  using two layers of 2D convolutions of filter size  $3 \times 3$  and stride 2 (Dong et al., 2018). The 6-layer decoder has no causal attention masks. Pre-norm residual paths are used (Nguyen and Salazar, 2019).
Training. The encoder and decoder could be trained separately (e.g. a refinement model that improves a pre-trained CTC model). In practice,
we found that joint training was essential for the decoder to learn non-identity behavior (as it is trained on the initial encoder hypothesis and its own iterations, with no further-noising). We apply dropout with  $p = 0.1$  on WSJ and  $p = 0.2$  on LibriSpeech. The initial CTC is weighted with  $\lambda = 0.3$ ; remaining weights are spread over the  $K$  iterations, with  $w_{1}$  three times larger than the rest. For WSJ, we used a batch size of 32 sequences. For LibriSpeech, we used a batch size of 25000 tokens. Gradients are accumulated in both cases. LibriSpeech models are trained over four V100 GPUs (p3.8xlarge instances on AWS). We trained to convergence on WSJ, and for 125K steps on LibriSpeech. Following Mask-CTC, we use label smoothing of 0.1 and the standard Transformer inverse square-root learning rate schedule with a linear warmup of 25000 steps. We double their transformer-1r factor to 10.0.
Decoding. At decode time, we average model weights over 30 training checkpoints. All RTFs are measured on a single CPU thread (-nj 1, ngpu=0). We take real-time duration (time) and divide by the total utterance duration in ESPnet's utt2dur file on eval92 for WSJ and test-other for LibriSpeech.
Tuning. Hyperparameters were set by manual tuning, with hyperparameters chosen based on performance on the validation set.
We validate our method on two standard benchmarks in speech recognition:
WSJ (Paul and Baker, 1992) is a dataset of spoken English text derived from articles in the Wall Street Journal from 1987-1989. The training set, SI-284, consists of 81 hours of data; we validated on the dev93 split (1.1 h) and evaluated on the eval92 split (0.7 h). The dataset is downloadable at the following links: https://catalog.ldc.upenn.edu/LDC93S6A and https://catalog.ldc.upenn.edu/LDC94S13A.
LibriSpeech (Panayotov et al., 2015) is a dataset of spoken English text derived from audiobooks from the LibriVox project. The training set consists of 1000 hours of data. The validation and test sets are split by speaker into clean and other splits based on the performance of an acoustic model trained on
WSJ. We validated on dev-clean (5.4 h) and evaluated on the test-clean (5.4 h) and test-other (5.1 h) splits. The dataset is downloadable at the following link: http://www.openslr.org/12/.