SlowGuess/NAACL_2021_OCR huggingface dataset:
Motoi Omachi and Yuya Fujita
Yahoo Japan Corporation
Shinji Watanabe and Matthew Wiesner
Center for Language and Speech Processing
Jhons Hopkins University
We propose a Transformer-based sequence-to-sequence model for automatic speech recognition (ASR) capable of simultaneously transcribing and annotating audio with linguistic information such as phonemic transcripts or part-of-speech (POS) tags. Since linguistic information is important in natural language processing (NLP), the proposed ASR is especially useful for speech interface applications, including spoken dialogue systems and speech translation, which combine ASR and NLP. To produce linguistic annotations, we train the ASR system using modified training targets: each grapheme or multi-grapheme unit in the target transcript is followed by an aligned phoneme sequence and/or POS tag. Since our method has access to the underlying audio data, we can estimate linguistic annotations more accurately than pipeline approaches in which NLP-based methods are applied to a hypothesized ASR transcript. Experimental results on Japanese and English datasets show that the proposed ASR system is capable of simultaneously producing high-quality transcriptions and linguistic annotations.
End-to-end automatic speech recognition (E2E ASR), which transcribes speech using a single neural network (NN), has recently gained traction (Graves and Jaitly, 2014; Chorowski et al., 2015; Chan et al., 2016; Graves, 2012; Dong et al., 2018). Existing E2E ASR models generate audio transcripts by sequentially producing likely graphemes, or multi-graphemic units, from which lexical items of a language can be recovered. However, other linguistic annotations such as phonemic transcripts, part-of-speech (POS) tags, or word boundaries, help understand the underlying audio characteristics (Simonnet et al., 2017). Such linguistic annotations are especially important in natural language processing (NLP) tasks done on audio
(a) One-to-many model.
(b) One-to-one model with a conditional chain mapping.
(c) One-to-one model with a single sequence.
Figure 1: Example of E2E ASR predicting two types of sequences.  $y_{m,l}$  denotes the  $m$ -th token of the  $l$ -th type of the sequence.
data, including spoken dialogue systems (Jurafsky and Martin, 2008). This study aims to endow existing E2E ASR models with the ability to produce such linguistic annotations.
Prior work explored using E2E ASR systems to predict multiple kinds of labels. Fig. 1 shows a diagram of these systems. These approaches use one of the following models: a one-to-many (O2M) model (Kubo and Bacchiani, 2020; Ueno et al., 2018; Gowda et al., 2019; Sanabria and Metze, 2018; Adams et al., 2019), a one-to-one (O2O) model with a conditional chain mapping (Shi et al., 2020), or an O2O model with a single sequence (Audhkhasi et al., 2018; Ghannay et al., 2018; Shafey et al., 2019; Yadav et al., 2020).
In O2M models shown in Fig. 1(a), a multitask objective is used in which an extra branch is tasked with estimating the secondary label sequence. For example, in (Kubo and Bacchiani, 2020), the phonemic transcript is produced in addition to the graphemic transcript. The O2M model can estimate each sequence more accurately than
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1861-1871 June 6-11, 2021. ©2021 Association for Computational Linguistics
separate models responsible for producing phonemic and graphemic transcripts independently. We can implement this approach with less effort by attaching multiple loss functions to the base architecture. However, this O2M model does not explicitly consider dependencies between phonemic and graphemic transcripts. Furthermore, aligning phoneme and grapheme sub-sequences requires additional post-processing based on time alignment or alignment across the multiple sequences during inference. Performance of downstream NLP tasks built on top of ASR outputs will suffer if this post-processing fails to generate alignment.
Fig. 1(b) shows an O2O model with a conditional chain mapping. This method for multiple sequence modeling has been applied to dialog modeling (Liang et al., 2020), speaker diarization (Fujita et al., 2020a), and multi-speaker ASR (Shi et al., 2020). Unlike the O2M model, this model can predict a variable number of output sequences while explicitly considering dependencies between the multiple sequences based on the probabilistic chain rule. However, modeling these inter-sequence dependencies requires more complicated neural architectures, and alignment of the sequences still requires post-processing during inference.
Another option for using O2O models is to output multiple sequences as a single sequence instead of using conditional chain mapping, as shown in Fig. 1(c). For example, in (Audhkhasi et al., 2018), the O2O model produces word transcripts by first generating a word's constituent graphemes followed by the word itself. Another application, explored in (Shafey et al., 2019) used the O2O model to produce graphemes followed by speaker role. This approach is the simplest to implement because we can reuse the neural network architecture used to produce the primary sequence to sequence mapping to produce the secondary label sequence (e.g., connectionist temporal classification (CTC) based systems). In contrast to the previous two approaches, the O2O model does not require postprocessing to align the label sequences during inference since the output sequence preserves the alignment between the word and corresponding annotation labels; alignment is only needed for the data preparation stage during training to produce the appropriate target sequences. For this reason, we used the O2O model in this study.
This paper proposes to use a state-of-the-art Transformer-based E2E ASR system (Karita et al.,
2019) for the O2O model with a single sequence, instead of CTC-based approaches which are frequently supported (Audhkhasi et al., 2018; Ghannay et al., 2018). Compared with the CTC-based systems, this approach can explicitly model the relationship between the output labels thanks to the autoregressive decoder network, similar to the conditional chain rule model in Fig. 1(b). We also demonstrate improved performance compared to the CTC-based systems. Another contribution is that we conducted an extensive empirical evaluation to analyze and demonstrate the utility of our approach. For example, we applied the method to English and Japanese ASR tasks in which phonemic transcripts and POS tags are simultaneously produced. Our approach predicts linguistic annotations correctly even though corresponding graphemes are wrong, while the pipeline approach, in which NLP-based methods are applied to a hypothesized ASR transcript, fails. This feature is helpful for the downstream NLP system like slot filling or intent detection. Besides, our approach is suitable for on-device applications because the E2E model archives small-footprint prediction (Pang et al., 2018). Note that our primary goal is to provide aligned transcripts and linguistic annotations with minimal degradation in ASR performance. We are not aiming to improve ASR performance. The features of the proposed method are summarized as follows:
- The proposed Transformer-based O2O model can explicitly model the relationship between the output graphemes and corresponding linguistic annotations, unlike the O2M and CTC-based O2O models.
- Our approach does not require additional alignment post-processing across the transcriptions and the sequence of the linguistic annotations during inference.
- We can easily combine the proposed O2O model with downstream NLP tasks and also conduct an intuitive error analysis (e.g., detecting the error caused due to the homonym by checking the word and the corresponding phoneme output).
2 Existing E2E ASR system
2.1 E2E ASR
The objective of E2E ASR is to estimate the output token sequence  $\mathbf{y} = \{y_{m}\in \mathcal{V}\}_{m = 1}^{L}$  from input feature sequences  $\mathbf{X} = \{\mathbf{x}_i\in \Re^{D^{\mathrm{in}}}\}_{i = 1}^{I^{\mathrm{in}}}$ . Here,  $D^{\mathrm{in}}$
and  $I^{\mathrm{in}}$  denote the number of the dimension of the input feature and the length of the input sequence, respectively; and  $L$  and  $\mathcal{V}$  denote output sequence length and the token vocabulary. To predict the output token sequence, an NN is trained to maximize the following conditional likelihood objective function:
$$
\begin{array}{l} \mathcal {L} = \log p (\mathbf {y} | \mathbf {X}) \\ = \sum_ {m = 1} ^ {M} \log p \left(y _ {m} \mid y _ {1}, \dots , y _ {m - 1}, \mathbf {X}\right). \tag {1} \\ \end{array}
$$
During run-time, the ASR output  $\hat{\mathbf{y}}$  is predicted by
$$
\hat {\mathbf {y}} = \underset {\mathbf {y} \in \mathcal {Y} ^ {*}} {\arg \max } \log p (\mathbf {y} | \mathbf {X}), \tag {2}
$$
where  $\mathcal{Y}^*$  denotes a set of all possible hypotheses.
The Transformer (Vaswani et al., 2017) is a state-of-the-art NN architecture that can be used to maximize Eq. (1). The Transformer consists of two NNs: The Encoder network and the Decoder network. Let  $I^{\mathrm{emb}}$  and  $D^{\mathrm{emb}}$  be the sequence length and dimension of the acoustic embedding. The Encoder network generates a sequence of embeddings of the acoustic information  $\mathbf{E} = \{\mathbf{e}_i \in \Re^{D^{\mathrm{emb}}} \}_{i=1}^{I^{\mathrm{emb}}}$  from input feature sequences, i.e.  $\mathbf{E} = \operatorname{Encoder}(\mathbf{X})$ . The Decoder network predicts the output of the  $M$ -th step  $y_M$  given a sub-sequence, including the current output  $\bar{\mathbf{y}} = \{y_1, \dots, y_{M-1}\}$  and  $\mathbf{E}$ , i.e.  $y_M = \operatorname{Decoder}(\bar{\mathbf{y}}, \mathbf{E})$ . This conditional autoregressive modeling function is particularly important in this paper since it can explicitly model the relationship between output labels, unlike CTC.
2.2 E2E ASR to predict two or more sequences
This study aims to estimate a word/morpheme sequence and linguistic annotations, such as phonemic transcripts or POS tags, simultaneously. Here, we define the number of sequences including the subword sequence  $\mathbf{y}_1$  in Section 2.1 and additional linguistic annotation sequences as  $\mathbf{y}_2,\ldots ,\mathbf{y}_K$ . To predict both transcriptions and linguistic annotations, the NN is trained to maximize the following log-likelihood of the joint probability:
$$
\mathcal {L} = \log p \left(\mathbf {y} _ {1}, \dots , \mathbf {y} _ {K} \mid \mathbf {X}\right), \tag {3}
$$
where  $\mathbf{y}_k = \{y_{1,k},\dots ,y_{M_k,k}|y_{m,k}\in \mathcal{Y}_k\}$  denotes an  $M_{k}$ -length sequence of the  $k$ -th type of tokens or linguistic annotations and  $\mathcal{Y}_k$  denotes a set of the corresponding tokens or symbols. In the rest of this
subsection, we explain the following existing models to maximize Eq. (3): the O2M model trained with multi-task learning and the O2O model trained with the conditional chain mapping.
2.2.1 O2M model trained with multi-task learning
One frequently used NN architecture (Ueno et al., 2018; Gowda et al., 2019; Sanabria and Metze, 2018; Adams et al., 2019) that maximizes Eq. (3) is the O2M model trained with multi-task learning. Fig. 1(a) shows the architecture of the model. The O2M model outputs several types of sequences independently. In other words, multi-task learning is derived by assuming conditional independence of output token types for Eq. (3), as follows:
$$
\begin{array}{l} \mathcal {L} = \log \prod_ {k = 1} ^ {K} p \left(\mathbf {y} _ {k} \mid \underline {{\mathbf {y} _ {1} , \cdots , \mathbf {y} _ {k - 1}}}, \mathbf {X}\right) (4) \\ = \sum_ {k = 1} ^ {K} \sum_ {m = 1} ^ {M _ {k}} \log p \left(y _ {m, k} \mid \bar {\mathbf {y}} _ {1: m - 1, k}, \mathbf {X}\right), (5) \\ \end{array}
$$
where  $\bar{\mathbf{y}}_{1:m-1,k} = \{y_{1,k},\dots,y_{m-1,k}\}$  denotes a sub-sequence of the  $k$ -th type of tokens or linguistic annotations up to  $m-1$ . The line crossing part of Eq. (4) represents that the sequences  $\mathbf{y}_1,\dots ,\mathbf{y}_{k-1}$  are neglected by assuming conditional independence. The purpose of this study is to predict words/morphemes and aligned linguistic annotation jointly. Since the O2M model deals with different lengths of sequences, post-processing is needed to align the multiple sequences. Also, Eq. (4) shows that multi-task learning assumes conditional independence, but transcripts and linguistic annotations are often conditionally dependent. Hence the O2M model is not ideal for this study.
2.2.2 O2O model trained with conditional chain mapping
The O2O model trained with a conditional chain mapping (Fujita et al., 2020a; Shi et al., 2020) can also be used to maximize Eq. (3). Fig. 1(b) shows the architecture of this model. This model predicts the different sequence types sequentially, each time conditioning on all previously decoded sequence types  $1 \dots k - 1$ . Different from the multi-task training loss (Eq. (4)) used in the O2M model, the O2O conditional chain mapping model is trained to maximize the joint log-likelihood (Eq. (3)) via a recursive expansion of the probabilistic chain rule. This model does not require or assume conditional
independence between sequence types. Formally, the O2O model is trained to maximize the following loss function:
$$
\begin{array}{l} \mathcal {L} = \log \prod_ {k = 1} ^ {K} p (\mathbf {y} _ {k} | \mathbf {y} _ {1}, \dots , \mathbf {y} _ {k - 1}, \mathbf {X}) \\ = \sum_ {k = 1} ^ {K} \sum_ {m = 1} ^ {M _ {k}} \log p \left(y _ {m, k} \mid \bar {\mathbf {y}} _ {m - 1, k}, \bar {\mathbf {Y}} _ {1: k - 1}, \mathbf {X}\right), \tag {6} \\ \end{array}
$$
where  $\bar{\mathbf{Y}}_{1:k - 1} = \{\mathbf{y}_1,\dots ,\mathbf{y}_{k - 1}\}$  denotes  $(k-$  1) sequences. While this approach can explicitly model inter-sequence dependencies, it still requires post-processing to align the output sequences.
3 Proposed E2E ASR system
Fig. 1(c) depicts the proposed single sequence O2O E2E ASR model. The single sequence O2O model predicts the word/morpheme and the corresponding linguistic annotations simultaneously by regarding multiple sequences as a single sequence.
In the single sequence representation, the  $K$  output sequences are collapsed into a single sequence of  $S$  segments. The  $i$ -th segment  $\mathbf{s}_i$  consists of a fixed order of  $K$  jointly aligned sub-sequences. Let  $\bar{\mathbf{y}}_{i,k}$  be the  $i$ -th sub-sequence of the  $k$ -th type of tokens or annotations from index  $B(i,k)$  to  $E(i,k)$ , i.e.,  $\bar{\mathbf{y}}_{i,k} := \bar{\mathbf{y}}_{B(i,k):E(i,k),k}$ . Then,  $\mathbf{s}_i = (\bar{\mathbf{y}}_{i,1},\dots,\bar{\mathbf{y}}_{i,K})$  denotes the  $i$ -th variable-length segment composed of aligned graphemic and linguistic annotation sub-sequences. Equation 8 shows how the  $K$  sequences are collapsed into a single sequence of composed of segment  $\mathbf{s}_i$ .
$$
\begin{array}{l} \left( \begin{array}{c} \mathbf {y} _ {1} \\ \vdots \\ \mathbf {y} _ {K} \end{array} \right) = \left( \begin{array}{c} \bar {\mathbf {y}} _ {1, 1}, \dots , \bar {\mathbf {y}} _ {S, 1} \\ \vdots \\ \bar {\mathbf {y}} _ {1, K}, \dots , \bar {\mathbf {y}} _ {S, K} \end{array} \right) (7) \\ = \left(\mathbf {s} _ {1}, \dots , \mathbf {s} _ {S}\right). (8) \\ \end{array}
$$
To obtain  $\mathbf{s}_i$ , we use existing annotation tools or manual annotations to jointly align the training sets of the  $K$  output sequence types. These segments are used as training targets in an auto-regressive prediction task. In this way, our model implicitly learns to simultaneously predict and align  $K$  output sequences from an input  $\mathbf{X}$ . We discuss further details of the data preparation in Section 3.2.
Letting  $y_{i}^{*}$  denote elements of the collapsed single-sequence representation  $(\mathbf{s}_1,\dots ,\mathbf{s}_S)$ , the
joint log-likelihood (Eq. (3)) can be written as
$$
\begin{array}{l} \mathcal {L} = \log p (\mathbf {y} _ {1}, \dots , \mathbf {y} _ {K} | \mathbf {X}) \\ = \sum_ {m = 1} ^ {M ^ {*}} \log p \left(y _ {m} ^ {*} \mid y _ {1} ^ {*}, \dots , y _ {m - 1} ^ {*}, \mathbf {X}\right). \tag {9} \\ \end{array}
$$
Note that this form is almost equivalent to the single sequence objective function in Eq. (1) except for the variable  $y_{m}^{*}$  takes values from the union of the  $K$  symbol sets that represent the  $K$  output sequences and the length of this sequence  $M^{*} = \sum_{k=1}^{K} M_{k}$ , is the sum of the lengths of the  $K$  output sequences.
This framework has various benefits compared with the existing frameworks described in Section 2. Similar to the O2O model trained with the conditional chain mapping in Section 2.2.2, this framework does not assume the conditional independence between output labels and has the flexibility to model the dependency between words/morphemes and linguistic annotations. Related works are using the O2O model, e.g., (Yadav et al., 2020), but they are based on CTC and do not consider such an explicit output dependency. Also, the proposed method using Transformer can preserve a relationship between the word/morpheme and the corresponding linguistic annotations across the sequence based on the aligned representation  $\mathbf{s}_i$  in Eq. (8). Finally, this framework is equivalent to the original single-sequence objective function, and we can use an existing strong sequence-to-sequence model (transformer in this paper) without any modifications of the algorithm. The only process is to prepare the collapsed single sequence composed of  $\mathbf{s}_i$ , which is discussed in the next section.
3.2 Data preparation
This section describes how we prepare the collapsed single sequence composed of  $\mathbf{s}_i$  in Eq. (8). We explain this data preparation with both English (TED-LIUM release 2 (TEDLIUM2) (Rousseau et al., 2014)) and Japanese (corpus of spontaneous Japanese (CSJ) (Maekawa et al., 2000)) data as an example. The sequence type includes the graphemic and phonemic transcipts<sup>1</sup>, as well as the POS tags.
Fig. 2 shows how to obtain the target sequence. First, we predict sequences of phonemes and POS tags from the graphemic sequences using manually
<sup>1</sup>In the Japanese task, we used the kana character, a syllabic character, and this paper regards it as a phoneme.
Figure 2: Example of the target sequence for the proposed model. <Ph->, <Pos->, and // denote symbols of phonemes and graphemes, and the word boundary, respectively. (a) the graphemic sequence; (b) sequences of phonemes and POS tags; (c) sequences whose annotations are mapped into specific symbols; (d) sub-sequences of graphemes, phonemes, and POS tags; (e) target sequence; (f) target sequence applied byte-pair encoding (Kudo and Richardson, 2018).
annotated labels or annotation tools (Fig. 2(a),(b)). For the Japanese data, we use the annotation labels provided in the corpus. Note that some of the POS tags are estimated using a morphological analysis model. For the English data, we obtain these sequences from the pronunciation dictionary provided in the corpus and WordNet (Miller, 1998), respectively. Some words in the vocabulary have two or more pronunciations in the pronunciation dictionary. To obtain phoneme sequences, we randomly selected a single pronunciation per word from the candidate pronunciations. Since in WordNet,  $57\%$  of the words in the corpus are not annotated with the POS tags, we annotated these labels with the output of the POS tagging system (Loper and Bird, 2002). Next, we replaced these phonemes and POS tags with special symbols (Fig. 2(c)) to distinguish them from the grapheme symbols. Third, we split graphemic and linguistic annotation sequences at word boundaries and obtain sub-sequences  $(\bar{\mathbf{y}}_{i,k}$  in Eq. (8)) (Fig. 2(d)). Then sub-sequences are aggregated with the segments  $(s_i$  in Eq. (8)) and collapsed into the target sequence in the manner of Eq. (8) (Fig. 2(e)). For the English data, we applied byte-pair encoding (BPE) (Kudo and Richardson, 2018) to the collapsed target sequence (Fig. 2(f)).
4.1.1 E2E ASR
We built a Transformer-based ASR system using the ESPnet toolkit (Watanabe et al., 2018). The
Transformer architecture and hyper-parameters for training/decoding are based on existing recipes in ESPnet. We investigated three models: self-attention-based CTC (Pham et al., 2019), the Transformer (Dong et al., 2018), and a hybrid Transformer trained with an auxiliary CTC objective (Transformer+CTC) (Karita et al., 2019). The CTC model was used in prior studies based on O2O models, e.g., (Audhkhasi et al., 2018; Yadav et al., 2020). During training, the CTC model was regularized with the Transformer decoder in the multitask learning fashion similar to Transformer+CTC. Such regularization techniques yield a significant improvement over a pure CTC baseline (Fujita et al., 2020b).
For the training of Transformer+CTC, we applied joint CTC training to improve performance (Karita et al., 2019). For CTC-based decoding, we used the greedy search algorithm. For Transformer decoding, we used the beam search algorithm and tuned search parameters using the development set. For the Transformer+CTC model, we applied Transformer/CTC joint decoding (Karita et al., 2019). and tuned the weights of the objective using the development set. Note that the language model shallow fusion (Hori et al., 2018) is not applied since we could not find effectiveness in our preliminary experiment.
4.1.2 Evaluation criteria
We evaluate the performance of the proposed method using the character error rate (CER), phoneme error rate (PER), and word error rate
(WER). CER and WER measure the quality of graphemic transcripts in Japanese and English respectively. PER is used to evaluate the quality of phonemic transcripts in both languages. This study aims to incorporate linguistic annotation prediction into the state-of-the-art Transformer-based E2E ASR. We computed the CER/WER/PER to verify that the E2E model can perform ASR adequately even though the additional downstream NLP tasks are incorporated.
To obtain a sequence with alignment  $(\mathbf{s}_i$  in Eq. (8)) on the inference stage, grapheme, phoneme, and POS should be generated in the same order as the training stage. To confirm this, we define annotation structure accuracy (ASA) as a metric. We can compute the correct number of the predicted structure and compute the accuracy. For example, the correct order of the output must follow the following grapheme-phoneme-POS order:
I <Ph12> <Pos3> go <Ph21> <Pos5> 
 where 
 and 
 denote the start and end symbols of a sentence, respectively. However, our sequence-to-sequence model does not have such explicit output constraints and it possibly outputs the following wrong order of the sequence:
$$
<   \mathrm {s} > \mathrm {I} <   \mathrm {P h} 1 2 > <   \mathrm {P o s} 3 > \mathrm {g o} <   \mathrm {P o s} 5 > <   / \mathrm {s} >
$$
Thus, the second case has 5 correct transition counts among 6 total transition counts, and we can compute the accuracy as  $5 / 6$ . We assume the transition from "go" to  $\langle \mathrm{Pos}5\rangle$  is incorrect.
To evaluate Japanese ASR's word segmentation performance, we measure the precision  $p$ , recall  $r$ , and F-value  $f$  of the hypothesized segmentation compared to the ground-truth segmentation. Let  $N^{\mathrm{hyp}}$ ,  $N^{\mathrm{ref}}$ , and  $N^{\mathrm{cor}}$  be the numbers of the predicted graphemes, the graphemes of the reference, and the graphemes whose predicted linguistic annotation is correct, respectively. The precision  $p$ , recall  $r$ , and F-value  $f$  are defined as follows:  $p = N^{\mathrm{cor}} / N^{\mathrm{hyp}}$ ,  $r = N^{\mathrm{cor}} / N^{\mathrm{ref}}$ ,  $f = 2pr / (p + r)$ . We only compared 1,919 utterances whose reference and hypothesis transcripts are exactly matched in order to ignore the effect of the ASR errors.
Additionally, hypothesized ASR transcripts and reference transcripts are aligned with graphemes, and we computed an annotation accuracy to measure the performance of the linguistic annotation. Let  $N^{\mathrm{in}}$  and  $N^{\mathrm{cor}}$  be the number of input words whose estimated grapheme is correct and the words whose estimated grapheme and linguistic annota