SlowGuess/NAACL_2021_OCR huggingface dataset:
tions are correct, respectively. The accuracy is computed by  $N^{\mathrm{cor}} / N^{\mathrm{in}}$ . Since we do not deal with the words whose grapheme is predicted incorrectly by ASR for computing the annotation accuracy, the annotation accuracy is robust to the ASR error.
Since the above measures for the word segmentation and linguistic annotation do not consider the ASR errors, we finally computed the following measures using all of the utterances (i.e., including ASR errors): normalized edit distance, precision, recall, and F-values.
4.1.3 Baseline of the linguistic annotation
To compare the linguistic annotation performance, we prepared a pipeline system, i.e., ASR followed by an NLP-based linguistic annotation. In the pipeline system, the separated model of CTC+Transformer first predicts graphemic sequences. Then, the linear SVM with L2 normalization, trained using KyTea (Graham and Mori, 2010), predicts word boundaries and linguistic annotation from the predicted sequences. To train KyTea, we only used the transcriptions in the ASR training set to perform a fair comparison to the proposed method.
The pipeline system for the Japanese task requires word segmentation before predicting linguistic annotations. The proposed ASR, on the other hand, achieves word segmentation and linguistic annotations simultaneously. Additionally, the proposed ASR achieves these estimates using graphemic information and acoustic information, but the pipeline system uses only the graphemic information. Hence, we expect that the proposed method can predict better word boundary and linguistic annotations for the sentence, which is hard to estimate only from graphemic information. Besides, our model might predict linguistic annotations correctly even though its transcripts are mispredicted, while the pipeline approach fails to predict linguistic annotation when the hypothesized ASR transcriptions include ASR errors. It is helpful for the downstream NLP-based system like slot filling or intent detection.
4.2 Performance of speech recognition
We evaluated ASR performance to confirm the proposed method can produce high-quality transcriptions and linguistic annotations. Note that our primary goal is to simultaneously predict transcription and linguistic annotations by keeping sufficient performance, not improving the ASR performance it-
outputs	model	CSJ	TEDLIUM2
eval1	eval2	eval3	dev	test
CER	PER	CER	PER	CER	PER	WER	PER	WER	PER
graphemes, phonemes	CTC (baseline)	7.4	5.0	5.5	3.1	5.9	3.3	15.7	7.3	15.6	7.7
Transformer	6.9	4.4	4.7	2.6	6.1	3.7	15.8	9.3	15.0	9.1
Transformer+CTC	6.1	3.8	4.3	2.3	4.6	2.5	10.3	4.9	9.3	4.7
graphemes, phonemes,POS	CTC (baseline)	10.0	7.0	7.3	4.4	8.3	5.1	15.8	7.2	14.9	7.0
Transformer	6.4	4.1	4.7	2.7	5.2	3.0	14.6	8.8	13.5	8.2
Transformer+CTC	6.7	4.3	4.9	2.7	5.3	2.9	10.3	4.7	9.5	4.7
Table 1: Comparison between CTC models, Transformer models, and Transformer+CTC models. CER, PER and WER denote character error rate, phoneme error rate, and word error rate, respectively. The CTC model was extensively used in prior work (e.g., Ghannay et al., 2018). Transformer+CTC refers to the Transformer with joint CTC training and decoding (Watanabe et al., 2018).
(a) CER/WER
(b) PER
Figure 3: Comparison between separated model and proposed joint modeling. The separated model predicts graphemes or phonemes; the joint model predicts graphemes and linguistic annotations simultaneously.
self. Table 1 and Fig. 3 show the ASR performance of the Japanese (CSJ) and English (TEDLIUM2) tasks.
First, we discuss which model architecture is appropriate for predicting the grapheme and phoneme sequences. Table 1 shows the Transformer or Transformer+CTC achieves better performance compared to the CTC model, which corresponds to the conventional method. This means that the Transformer is better for predicting transcriptions and linguistic annotations (phoneme in this experiment) than CTC thanks to the explicit dependency modeling, as discussed in Section 3. Since Transformer+CTC yields better or equivalent performance than the Transformer, we used Transformer+CTC architecture as a base model in the rest of this paper (refer to as a joint model).
Second, we discuss whether the proposed joint models predict the grapheme and phoneme with sufficient performance. To confirm that, we trained two separate models, which predict either a grapheme sequence or a phoneme sequence. Since Transformer+CTC yields better performance than the CTC model and Transformer, we used Transformer+CTC architecture as a base model. Fig. 3
shows that the proposed joint model is almost comparable to the separated model, especially when it predicts both graphemes and phonemes. When the joint model prediction includes the POS tag, we observed a slight degradation, especially in the Japanese task. However, such degradation is still less than  $1\%$ , and we can conclude the proposed O2O model of Transformer+CTC can predict graphemes and phonemes simultaneously with sufficient performance. We would emphasize that the proposed joint model can have alignment between grapheme/phoneme/POS while the conventional separated model can not.
4.3 Performance of the annotation structure prediction
As we discussed in Section 4.1.2, we computed the annotation structure accuracy (ASA), and it turns out that its range was from  $98.9\%$  to  $100.0\%$ . This means that the proposed joint model can consistently predict transcriptions and the linguistic annotations in the correct order almost perfectly. We found that almost all errors of the transition occurred in the last word, which might be caused by beam search errors.
(a) Transcription including repetition.
Reference: その/sono/adnominal 後/go/noun 音楽/oNgaku/noun 番組/baNgumi/noun が/ga/noun 全盛/zeNse:/noun ... Pipeline : その/sono/adnominal 後/ato/noun 音楽/oNgaku/noun 番組/baNgumi/noun が/ga/noun 全盛/zeNse:/noun ... Proposed: その/sono/adnominal 後/go/noun 音樂/oNgaku/noun 番組/baNgumi/noun が/ga/noun 全盛/zeNse:/noun ...
(b) Transcription including heteronym.
Figure 4: Examples of the estimated transcription. X, in X/Y/Z, denotes graphemes, and Y, Z denotes phonemes and POS tags, respectively.
System	Precision	Recall	F-value
Pipeline	99.6	99.5	99.6
Proposed	99.8	99.8	99.8
Table 2: Word segmentation performance in CSJ (\%). Pipeline is ASR predicting transcriptions followed by the NLP-based linguistic annotation system (Graham and Mori, 2010). Proposed predicts graphemes and phonemes followed by POS tags from speech. Note that we used only the sentences whose hypothesized ASR transcript is predicted correctly for evaluation.
System	CSJ	TEDLIUM2
Phoneme	POS	Phoneme	POS
Pipeline	99.4	99.0	99.8	99.7
Proposed	99.8	99.4	99.9	99.9
4.4 Performance of word segmentation and predicting linguistic annotations
We evaluated the performance of word segmentation and linguistic annotations using the output of the proposed ASR, which predicts graphemes, phonemes, and POS tags. Note that we did not compute the word segmentation performance in the English task because the English sentences include word boundaries.
Tables 2 and 3 show the performance of the word segmentation and of the predicting linguistic annotations, respectively. Note that these results do not consider the ASR error. These tables show that the proposed ASR system achieves better word segmentation and predicts linguistic annotations
Table 3: Accuracy of predicting linguistic annotation  $(\%)$ . Note that we removed the graphemes which do not appear in the reference from the evaluation to ignore the effect of the ASR errors.
System	CSJ	TEDLIUM2
Phoneme	POS	Phoneme	POS
Pipeline	0.08	0.06	0.10	0.08
Proposed	0.08	0.05	0.10	0.07
Table 4: Normalized edit distance averaged over the whole evaluation set. Note that we consider the effect of the ASR errors.
better than the pipeline system. To compute accuracy, we used 41k and 65k morphemes for CSJ and TEDLIUM2, respectively, and we consider the number of samples is enough to show our model is better than the pipeline approach.
Table 4 and 5 show the performance which considers ASR errors. Table 4 shows that the proposed ASR system achieves better prediction of the POS tags than the pipeline, even though the proposed system sometimes failed to predict the transcriptions<sup>2</sup>. Table 5 also confirms that the proposed ASR system predicts better performance in the Japanese task. Although the performance of the pipeline system and the proposed ASR system is comparable in the English task, we would like to emphasize that the proposed ASR does not require extra memory for the additional downstream NLP task. This is useful for developing a small footprint system.
Fig. 4 shows some examples that the proposed ASR can estimate the word boundary and phonemes correctly. For example, the first sentence correctly segments the word boundary based on the "repetition" POS tag estimated from the acoustic information. Similarly, the second sentence appropriately chooses the correct pronunciation from the acoustic information.
5.1 Building pronunciation dictionary
Since our system of E2E ASR can estimate pairs of graphemes and phonemes for each word, we can build a pronunciation dictionary by considering both graphemes to phoneme sequences and acoustic information.
Table 6 shows the entries of the pronunciation dictionary extracted from the output of our system. The first row of the table lists the entries
Reference: ヒツチ/piqchi/noun と/to/particle スυケトラ/supekutora/repetition スυケTRL/supekutoru/noun 包絡/ho:raku/noun ... Pipeline： ヒツチ/piqchi/noun と/to/particle スυケTRL/SUKETUL/UNK/UNK 包絡/ho:raku/noun ... Proposed: ヒツチ/piqchi/noun と/to/particle スυケTRL/supekutora/repetition スυケTRL/supekutoru/noun 包絡/ho:raku/noun ...
2We conducted Welch's t-test and found a significant difference between the POS values of the pipeline system and the proposed ASR system  $(p < 0.01)$
System	CSJ	TEDLIUM2
Precision	Recall	F-value	Precision	Recall	F-value
Pipeline	79.4	82.6	81.4	84.5	71.5	77.5
Proposed	85.0	85.6	85.3	82.6	71.2	76.5
(a) Phoneme
System	CSJ	TEDLIUM2
Precision	Recall	F-value	Precision	Recall	F-value
Pipeline	79.1	82.6	80.8	84.5	58.3	69.0
Proposed	84.9	84.8	84.8	83.2	58.3	68.6
(b) POS
Table 5: Precision, recall, and F-values of the linguistic annotation prediction. We used all of the sentences, including the hypothesized ASR transcript including the ASR error, for evaluation.
Entries
(a)	REF	kindergarteners/K,IH,N,D,ER,G,AA,R,T,AH,N,ER,Z	overcooked/OW,V,ER,K,UH,K,T	jovial/JH,OW,V,IAH,L
HYP	kindergarteners/K,IH,N,D,ER,G,AA,R,T,AH,N,ER,Z	overcooked/OW,V,ER,K,UH,K,T	jovial/JH,OW,V,IAH,L
(b)	REF	Himalaya/HH,IH,M,AH,L,AY,AH	forest/F,AO,R,IH,S,T	object/AA,B,JH,EH,K,T
HYP	Himalaya/HH,IH,M,AH,L,ey,AH	forest/F,AO,R,AH,S,T	object/AH,B,JH,EH,K,T
Table 6: Entries of the pronunciation dictionary generated using the output of the proposed E2E ASR. X and Y, in X/Y, denote transcription and phoneme sequence, respectively; and REF and HYP denote reference and hypothesis, respectively. The row of (a) and (b) list the entries of the out-of-vocabulary word and the entries of the heteronym, respectively.
whose words did not appear in the text of the training set and whose phoneme sequence is estimated correctly. These entries indicate that our system can predict the phonemes of OOV words. The second row of the table shows the entries whose phonemes are different from the reference but exist in the CMU pronunciation dictionary (CMU). In other words, these entries have variations of the phoneme sequence for each word, and the phoneme sequences are predicted correctly. In this study, we removed the phoneme sequence variations for each grapheme from the training set. If the Transformer is trained to predict phoneme sequences using only linguistic information, the phoneme sequences are likely to be mapped into words deterministically. Interestingly, our Transformer recovers the variations of the phoneme sequences for each word. It seems that the acoustic information contributed to predicting the phoneme sequences.
5.2 Attention pattern
One of the Transformer's additional benefits is that we can deduce what is happening inside the Transformer by visualizing the patterns of self-attention and source-target attention weights.
Fig. 5 depicts patterns of the self-attention and source-target attention weights on the third layer of the Decoder network. This figure shows that self-attention changes monotonically but has additional diagonal dotted lines. This means that self-attention uses the multiple (both grapheme and
(a) Self attention
(b) Source-target attention
Figure 5: Example of the attention pattern of the third layer of the Decoder network. The transformer estimates graphemes and phonemes simultaneously.
phoneme) output symbols but mostly preserving the order of the sequence. Similarly, it also shows that source-target attention focuses on the acoustic feature of the same time step twice. It shows that both graphemes and phonemes are predicted using the same acoustic features at the same time step, respectively.
We proposed a novel E2E ASR Transformer system for simultaneously estimating transcriptions and linguistic annotations such as phonemic transcripts or POS tags. This paper showed that the proposed ASR could estimate these features with sufficient performance and also showed reasonable phoneme and grapheme analyses and attention patterns thanks to the aligned output of both output symbols. In future work, we will extend the proposed approach to predict other linguistic annotations such as named entities.
The Carnegie mellon pronouncing dictionary. [Online]. Available: "http://www.speech.cs.cmu.edu/cgi-bin/cmudict [accessed 21-Aug-2020].
O. Adams, M. Wiesner, S. Watanabe, and D. Yarowsky. 2019. Massively multilingual adversarial speech recognition. In Proc. ACL, pages 96-108.
K. Audhkhasi, B. Kingsbury, B. Ramabhadran, G. Saon, and M. Picheny. 2018. Building competitive direct acoustics-to-word models for english conversational speech recognition. In Proc. ICASSP, pages 4759-4763.
W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals. 2016. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In Proc. ICASSP, pages 4960-4964.
J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio. 2015. Attention-based models for speech recognition. In NIPS, volume 28.
L. Dong, S. Xu, and B. Xu. 2018. Speech-Transformer: A no-recurrence sequence-to-sequence model for speech recognition. In Proc. ICASSP, pages 5884-5888.
Y. Fujita, S. Watanabe, S. Horiguchi, Y. Xue, J. Shi, and K. Nagamatsu. 2020a. Neural speaker diarization with speaker-wise chain rule. arXiv:2006.01796.
Y. Fujita, S. Watanabe, M. Omachi, and X. Chang. 2020b. Insertion-Based Modeling for End-to-End Automatic Speech Recognition. In Proc. Interspeech, pages 3660-3664.
S. Ghannay, A. Caubrière, Y. Esteve, N. Camelin, E. Simonnet, A. Laurent, and E. Morin. 2018. End-to-end named entity and semantic concept extraction from speech. In Proc. SLT, pages 692-699.
D. Gowda, A. Garg, K. Kim, M. Kumar, and C. Kim. 2019. Multi-task multi-resolution char-to-BPE cross-attention decoder for end-to-end speech recognition. In Proc. Interspeech, pages 2783-2787.
N. Graham and S. Mori. 2010. Word-based partial annotation for efficient corpus construction. In Proc. LREC, pages 2723-2727.
A. Graves. 2012. Sequence transduction with recurrent neural networks. arXiv:1211.3711.
A. Graves and N. Jaitly. 2014. Towards end-to-end speech recognition with recurrent neural networks. In Proc. ICML, volume II, pages 1764-1772.
T. Hori, J. Cho, and S. Watanabe. 2018. End-to-end speech recognition with word-based RNN language models. In Proc. SLT, pages 389-396.
D. Jurafsky and J. Martin. 2008. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, volume 2.
S. Karita, N. E. Y. Soplin, S. Watanabe, M. Delcroix, A. Ogawa, and T. Nakatani. 2019. Improving Transformer-based end-to-end speech recognition with connectionist temporal classification and language model integration. In Proc. Interspeech, pages 1408-1412.
Y. Kubo and M. Bacchiani. 2020. Joint phoneme-grapheme model for end-to-end speech recognition. In Proc. ICASSP, pages 6119-6123.
T. Kudo and J. Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. In Proc EMNLP, pages 66-71.
W. Liang, Y. Tian, C. Chen, and Z. Yu. 2020. MOSS: End-to-end dialog system framework with modular supervision. 34(05):8327-8335.
E. Loper and S. Bird. 2002. NLTK: the natural language toolkit. Proc. ETMTNLP, page 63-70.
K. Maekawa, H. Koiso, S. Furui, and H. Isahara. 2000. Spontaneous speech corpus of Japanese. In Proc. of LREC, pages 946-952.
G. A Miller. 1998. WordNet: An electronic lexical database. MIT press.
R. Pang, T. Sainath, R. Prabhavalkar, S. Gupta, Y. Wu, S. Zhang, and C.-C. Chiu. 2018. Compression of end-to-end models. In Proc. Interspeech, pages 27-31.
N.-Q. Pham, T.-S. Nguyen, J. Niehues, M. Müller, and A. Waibel. 2019. Very deep self-attention networks for end-to-end speech recognition. Proc. Interspeech, pages 66-70.
A. Rousseau, P. Deleglise, and Y. Esteve. 2014. Enhancing the TED-LIUM corpus with selected data for language modeling and more TED talks. In Proc. LREC, pages 3935-3939.
R. Sanabria and F. Metze. 2018. Hierarchical multitask learning with CTC. In Proc. SLT, pages 485-490.
Laurent El Shafey, Hagen Soltau, and Izhak Shafran. 2019. Joint Speech Recognition and Speaker Diarization via Sequence Transduction. In Proc. Interspeech, pages 396-400.
J. Shi, X. Chang, P. Guo, S. Watanabe, Y. Fujita, J. Xu, B. Xu, and L. Xie. 2020. Sequence to multisequence learning via conditional chain mapping for mixture signals. In Proc. NIPS, volume 33, pages 3735-3747.
E. Simonnet, S. Ghannay, N. Camelin, Y. Esteve, and R. D. Mori. 2017. ASR error management for improving spoken language understanding. In Proc. Interspeech, pages 3329-3333.
S. Ueno, H. Inaguma, M. Mimura, and T. Kawahara. 2018. Acoustic-to-word attention-based model complemented with character-level CTC-based model. In Proc. ICASSP, pages 5804-5808.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.-N. Gomez, L. Kaiser, and I. Polosukhin. 2017. Attention is all you need. In NIPS, pages 5998-6008.
S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N. Enrique Yalta Soplin, J. Heymann, M. Wiesner, N. Chen, A. Renduchintala, and T. Ochiai. 2018. ESPnet: End-to-end speech processing toolkit. In Proc. Interspeech, pages 2207-2211.
H. Yadav, S. Ghosh, Y. Yu, and R. R. Shah. 2020. End-to-End Named Entity Recognition from English Speech. In Proc. Interspeech, pages 4268-4272.