SlowGuess/NAACL_2021_OCR huggingface dataset:
Yu-An Chung $^{1*}$ , Chenguang Zhu $^{2*}$ , Michael Zeng $^{2}$
<sup>1</sup>MIT Computer Science and Artificial Intelligence Laboratory
2Microsoft Cognitive Services Group
andyuyuan@mit.edu,{chezhu,nzeng}@microsoft.com
Spoken language understanding (SLU) requires a model to analyze input acoustic signal to understand its linguistic content and make predictions. To boost the models' performance, various pre-training methods have been proposed to learn rich representations from large-scale unannotated speech and text. However, the inherent disparities between the two modalities necessitate a mutual analysis. In this paper, we propose a novel semi-supervised learning framework, SPLAT, to jointly pre-train the speech and language modules. Besides conducting a self-supervised masked language modeling task on the two individual modules using unpaired speech and text, SPLAT aligns representations from the two modules in a shared latent space using a small amount of paired speech and text. Thus, during fine-tuning, the speech module alone can produce representations carrying both acoustic information and contextual semantic knowledge of an input acoustic signal. Experimental results verify the effectiveness of our approach on various SLU tasks. For example, SPLAT improves the previous state-of-the-art performance on the Spoken SQuAD dataset by more than  $10\%$ .
Spoken language understanding (SLU) tackles the problem of comprehending audio signals and making predictions related to the content. SLU has been widely employed in various areas such as intent understanding (Tur and De Mori, 2011; Bhargava et al., 2013; Ravuri and Stolcke, 2015; Lusosch et al., 2019), question answering (Lee et al., 2018; Chuang et al., 2020), and sentiment analysis (Zadeh et al., 2018). Early approaches leverage a two-step pipeline: use automatic speech recognition (ASR) to transcribe input audio into text, and then employ language understanding models to produce
results. However, such cascaded system has several drawbacks. First, the transcription produced by the ASR module often contains errors, which adversely affects the language understanding module's prediction accuracy. Second, even if the transcription is perfect, the rich prosodic information of speech (e.g., tempo, pitch, and intonation) is inevitably lost after ASR. In comparison, humans often leverage these information to better understand and disambiguate the content. Therefore, there has been a rising trend of end-to-end approaches to retain information from audio signals to carry out the understanding task (Serdyuk et al., 2018; Chen et al., 2018; Haghani et al., 2018).
While end-to-end SLU methods are effective, they often suffer from a shortage of labeled training data, especially when the target task is in a novel domain. One solution is to leverage self-supervised training as is done in pre-trained language models. Examples like BERT (Devlin et al., 2019), GPT (Radford et al., 2018), and RoBERTa (Liu et al., 2019) are first pre-trained on large-scale unannotated text in a self-supervised fashion to learn rich textual representations before being finetuned on downstream tasks with a modest amount of labeled data. Borrowing this idea, several pretraining methods have been proposed for speech, e.g., wav2vec (Schneider et al., 2019; Baevski et al., 2020a), contrastive predictive coding (Oord et al., 2018; Rivière et al., 2020), autoregressive predictive coding (Chung et al., 2019a, 2020; Chung and Glass, 2020b), and DeCoAR (Ling et al., 2020; Ling and Liu, 2020), to capture contextual representations from unlabeled speech data. Nevertheless, these methods leverage only acoustic data and mainly focus on modeling the acoustic information during pre-training. As a result, the produced representations may not be optimal for language understanding tasks.
To solve these problems, we propose a novel Speech-Language joint pre-Training framework,
* Equal contribution. The work was done when Yu-An Chung was interning at Microsoft.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1897-1907 June 6-11, 2021. ©2021 Association for Computational Linguistics
Step 1: Speech module pre-training
Figure 1: Overview of SPLAT. First, the speech and language modules are separately pre-trained using speech and text data via masked language modeling (MLM). In practice, we directly employ the BERT<sub>BASE</sub> model released by Devlin et al. (2019) to be the language module. Then, by leveraging a small amount of paired speech and text data, either a sequence-level alignment loss  $\mathcal{L}_{seq}$  or a token-level alignment loss  $\mathcal{L}_{tok}$  is applied to align the representations from both modules in a shared latent space (only  $\mathcal{L}_{seq}$  is shown here). During alignment, the language module is kept frozen and only the speech module is updated. Before aligning the two modules, there is an optional step to update the BERT<sub>BASE</sub>-initialized language module via MLM using the text portion from the paired data. This optional step aims to adapt the language module to the speech domain to facilitate later alignment. After pre-training, the language module is discarded and only the speech module is used in downstream tasks.
SPLAT. SPLAT contains a speech module and a language module for multi-modal understanding. The speech module is a Transformer encoder trained from scratch and the language module is initialized from BERT. Both modules leverage large-scale unannotated data for pre-training via masked language modeling. In the speech module, each frame is seen as a token and is replaced with zero vector with a certain probability. For each masked frame, we minimize the L1-distance between the predicted frame and the original frame.
Then, to make the speech module aware of the contextual information extracted from the language module, we design an alignment loss to align the representations from both modules in a shared latent semantic space. In detail, we propose two alignment methods, a sequence-level one and a token-level one, that leverage a small amount of paired speech and text to minimize the disparity between the acoustic representations from the speech module and the textual representations from the language module. In this way, the speech representations will carry not only the acoustic information but also the contextual knowledge from the text. After this alignment, when text input is absent during
fine-tuning, the speech module alone can produce representations that bridge the speech input and the language understanding output.
We conduct extensive evaluations on several downstream SLU tasks, including Fluent Speech Commands for intent detection, Switchboard for dialog act classification, CMU-MOSEI for spoken sentiment analysis, and Spoken SQuAD for spoken question answering. SPLAT achieves superior results in all datasets. For example, SPLAT improves the previous state-of-the-art performance on the Spoken SQuAD dataset by more than  $10\%$ . Furthermore, we show that SPLAT can perform well even given just a tiny portion of the labeled training data in downstream tasks.
Spoken language understanding In recent years, due to its flexibility and effectiveness, end-to-end spoken language understanding (SLU) has been proposed and applied to various tasks (Qian et al., 2017; Serdyuk et al., 2018; Ludosch et al., 2019). For instance, Qian et al. (2017) use an autoencoder to initialize the SLU model. Ludosch et al. (2019) pre-train the model to recognize words and
phonemes, and then fine-tune it on downstream tasks. Chen et al. (2018) pre-train the model to categorize graphemes, and the logits are fed into the classifier. In most of these approaches, the model pre-training requires annotated speech, e.g., word or phonemes corresponding to audio signals. As a result, the massive unlabeled speech data cannot be utilized by these models.
Self-supervised pre-training for language Pre-trained models have achieved great success in both language and speech domains. In language, BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), UniLM (Dong et al., 2019), and BART (Lewis et al., 2020) have been successfully applied to natural language inference (Zhang et al., 2020b), question answering (Zhu et al., 2018), and summarization (Zhu et al., 2019). These pretrained models leverage self-supervised tasks such as masked language modeling (MLM), next sentence prediction, and de-noising autoencoder.
Self-supervised pre-training for speech In speech, wav2vec (Schneider et al., 2019) leverages contrastive learning to produce contextual representations for audio input; vq-wav2vec (Baevski et al., 2020a) and wav2vec 2.0 (Baevski et al., 2020b) further propose to discretize the original continuous audio signals in order to enable more efficient MLM training with Transformer (Vaswani et al., 2017). Pre-trained speech models have been applied to ASR (Ling et al., 2020; Chung and Glass, 2020a; Baevski et al., 2020b), phoneme recognition (Song et al., 2020; Liu et al., 2020a), speech translation (Nguyen et al., 2020; Chung et al., 2019c), and speech synthesis (Chung et al., 2019b), to name a few.
Nevertheless, an SLU model must incorporate both acoustic and language understanding capabilities to project speech signals to semantic outputs. Thus, a pre-trained model for SLU needs to address tasks beyond a single modality.
Speech and language joint pre-training. Recently, SLU applications have prompted joint pretraining on both speech and text data. SpeechBERT (Chuang et al., 2020) applies MLM to pairs of audio and transcripts. However, there are several crucial differences to compared to our work. First, SpeechBERT contains a phonetic-semantic embedding module that requires forced alignment to first segment speech into word segments to obtain. Second, both the pre-training and fine-tuning phases
of SpeechBERT require both speech and text input, since it is designed for a specific spoken question answering task. However, many SLU tasks only take speech as input, which does not align with the design of SpeechBERT. In contrast, our model can learn to align acoustic and textual representations using just (a small amount of) paired data during pre-training, and only needs speech input for downstream tasks.
Denisov and Vu (2020) propose to align speech and language embeddings in a method similar to ours. However, there are several key differences. First, Denisov and Vu (2020) employ the encoder of a pre-trained ASR model, which already requires plentiful of annotated speech to obtain. Our model, on the other hand, conducts self-supervised learning to pre-train the speech module using unannotated speech. Secondly, besides sequence-level alignment, we propose a token-level alignment method, which is suitable for token-level downstream tasks. Last but not least, our model uses a much smaller paired speech and text for alignment (10 hours) than Denisov and Vu (2020) (1,453 hours), yet still largely outperforms their method in intent detection and dialog act classification.
In this section we present SPLAT, a framework for learning joint contextual representations of speech and language. The model consists of a speech module and a language module that share a similar architecture and learning algorithm. The pretraining of SPLAT is divided into two steps. First, we individually pre-train the speech and language modules using unannotated speech and text, respectively. Then, we leverage a simple yet effective alignment task that uses only a small amount of paired speech and text data to align the representations from both modules in a shared latent semantic space such that the information learned by the language module is transferred to the speech module. After pre-training, the language module is discarded and only the speech module is used in downstream tasks.
Below we formally describe the procedures for pre-training the speech (§3.1) and language modules (§3.2), and the alignment loss (§3.3) for aligning the representations from the two modules. Figure 1 provides an overview of the pre-training procedures of SPLAT.
Figure 2: Token-level alignment between speech and language modules.  $(s_1, \dots, s_5)$  are the output embeddings of the speech module and  $(t_1, \dots, t_4)$  are those of the language module.
3.1 Speech module pre-training
The goal of this module is to leverage unlabeled speech data to learn representations that capture meaningful acoustic information about speech utterances such as their phonetic content and speaker characteristics. Formally, the input to the speech module is a 80-dimensional log Mel spectrogram,  $(\pmb{x}_1,\dots,\pmb{x}_n)$ , where  $\pmb{x}_i\in \mathbb{R}^{80},1\leq i\leq n$ . The speech module, which is implemented as a Transformer architecture, then produces hidden representations  $(s_1,\ldots ,s_n)$  and predictions  $(\hat{\pmb{x}}_1,\dots,\hat{\pmb{x}}_n)$  where  $s_i\in \mathbb{R}^{768}$  and  $\hat{\pmb{x}}_i\in \mathbb{R}^{80}$ .
To boost its capacity for contextual understanding, we borrow the idea of masked language modeling (MLM) (Devlin et al., 2019; Liu et al., 2020c; Wang et al., 2020; Liu et al., 2020b). Specifically, each audio frame  $\boldsymbol{x}_i$  is replaced with a zero vector with a probability of  $15\%$ . The corresponding output  $\hat{\boldsymbol{x}}_i$  is trained to be close to the original frame  $\boldsymbol{x}_i$  via minimizing their L1-distance. Additionally, since consecutive frames are highly correlated, it is possible that the model simply utilizes the local smoothness of speech signals for reconstructing a single frame and thus fails to capture useful information. To avoid such issue, when a frame  $\boldsymbol{x}_i$  is selected to be masked, its following three frames  $\boldsymbol{x}_{i+1}$ ,  $\boldsymbol{x}_{i+2}$ , and  $\boldsymbol{x}_{i+3}$  are also masked, and the model is asked to reconstruct all these masked frames.
Furthermore, according to SpecAugment (Park et al., 2019), the input features  $(\pmb{x}_1,\dots,\pmb{x}_n)$  can be seen as comprising two dimensions: time, i.e., the subscript  $i$ , and channel, i.e., the elements in each  $\pmb{x}_i$ . While conventional MLM masks along certain time steps, the input signals can also be masked along the channel dimension. In other words, each column vector  $[\pmb{x}_{1,j},\dots,\pmb{x}_{n,j}]$  for  $1\leq j\leq 80$  has a  $15\%$  of chance to be masked, i.e., replaced with a zero vector. This channel masking is
combined with temporal masking to reinforce the model's capability to utilize contextual information from both time and channel, and reduce the impact of co-adaptation between acoustic frames. The final pre-training objective for the speech module is to reconstruct the entire input sequence from the altered version of it:
$$
\mathcal {L} _ {s p} = \sum_ {i = 1, 2, \dots , n} \| \boldsymbol {x} _ {i} - \hat {\boldsymbol {x}} _ {i} \| _ {1} \tag {1}
$$
We use the speech portion of the train-clean-360 subset from the LibriSpeech corpus (Panayotov et al., 2015) to pre-train the speech module, i.e., to minimize  $\mathcal{L}_{sp}$ . This subset contains 360 hours of read speech produced by 921 speakers. We follow the standard Kaldi setting, using a frame size of  $25\mathrm{ms}$  and a time shift of  $10\mathrm{ms}$  for generating the 80-dimensional log Mel spectrograms. The spectrograms are normalized to zero mean and unit variance per speaker.
3.2 Language module pre-training
The language module aims to offer contextual understanding for text input. We directly employ the BERT<sub>BASE</sub> model released by Devlin et al. (2019), which is pre-trained on a large text corpus with the MLM task and contains rich textual representations, as the language module. We denote the cross-entropy loss for the language MLM task as  $\mathcal{L}_{text}$ .
Given input token embeddings  $(\pmb{y}_1, \dots, \pmb{y}_m)$ , where  $\pmb{y}_1$  corresponds to the [CLS] token, the module produces contextual representations  $(t_1, \dots, t_m)$ , where  $t_j \in \mathbb{R}^{768}, 1 \leq j \leq m$ .
3.3 Aligning speech and language representations
The input to most SLU tasks consists of only audio signals, but the model is required to conduct semantic understanding, which can be best handled when
textual information is present. Therefore, we propose to align the pre-trained speech and language representations in a shared semantic latent space.
Suppose a pair of speech and text data consisting of an acoustic feature sequence  $(\pmb{x}_1, \dots, \pmb{x}_n)$  and its transcript  $(\pmb{y}_1, \dots, \pmb{y}_m)$ . The speech and language modules separately produce the output representations  $(s_1, \dots, s_n)$  and  $(t_1, \dots, t_m)$ . We then propose two methods to align the embeddings from the modules: sequence-level and token-level alignment.
Sequence-level alignment For sequence-level alignment, we treat the first embeddings from the two output representations, i.e.,  $s_1$  and  $t_1$ , as the sequence-level representations of their respective sequences, and minimize their L1-distance:
$$
\mathcal {L} _ {\text {s e q}} = \left\| \boldsymbol {s} _ {1} - \boldsymbol {t} _ {1} \right\| _ {1} \tag {2}
$$
Since our goal is to transfer the textual knowledge contained by the language module to the speech module, we only update the speech module to minimize  $\mathcal{L}_{seq}$  and keep the language module fixed.
After pre-training, when the transcript is absent in downstream tasks, the first output embedding of the speech module  $s_1$  will still be close to its corresponding text embedding  $t_1$  from the language module, as if the transcript were given. It follows that  $s_1$  can then be used to predict the property of the whole audio input, e.g., intent classification.
Token-level alignment To achieve a finer level of alignment, each audio feature should be compared with its each text token. Although forced alignment (Gorman et al., 2011) can establish this correspondence between audio signals and individual words, it requires a pre-trained ASR system to obtain. Here we propose a method that automatically aligns audio features with textual tokens.
Inspired by BERTScore (Zhang et al., 2020a), for each output text embedding  $t_j$ , we first compute its cosine similarity with each output acoustic embedding  $s_i$ , and select the acoustic feature with the highest similarity. Then, the alignment is performed by maximizing the sum of these maximum similarities over all tokens, weighted by each token's inverse document frequency (idf) to reduce the impact of common words:
$$
\mathcal {L} _ {t o k} = - \frac {\sum_ {j = 1} ^ {m} \operatorname {i d f} \left(\boldsymbol {t} _ {j}\right) \max _ {i} \operatorname {c o s s i m} \left(\boldsymbol {s} _ {i} , \boldsymbol {t} _ {j}\right)}{\sum_ {j = 1} ^ {m} \operatorname {i d f} \left(\boldsymbol {t} _ {j}\right)} \tag {3}
$$
The token-level alignment loss is illustrated in Figure 2. Same as  $\mathcal{L}_{seq}$ , when minimizing  $\mathcal{L}_{tok}$ , the
Algorithm 1 Pre-training SPLAT
Input: An unlabeled speech corpus  $\mathcal{X} = \{\pmb{x}^{(p)}\}_{p=1}^{N}$ , an unlabeled text corpus  $\mathcal{Y} = \{\pmb{y}^{(q)}\}_{q=1}^{M}$ , and a paired speech-text corpus  $\mathcal{Z} = \{(\pmb{x}^{(k)}, \pmb{y}^{(k)})\}_{k=1}^{K}$ , where  $K \ll N, M$ .
1: Use  $\mathcal{X}$  to train the speech module by minimizing  $\mathcal{L}_{sp}$  (Equation 1).
2: Use  $\mathcal{V}$  to train the language module by minimizing  $\mathcal{L}_{text}$  (we directly employ BERT<sub>BASE</sub> from Devlin et al. (2019) for this step).
3: Use  $\{\pmb{y}^{(k)}\}_{k = 1}^{K}$  from  $\mathcal{Z}$  to train the language module by minimizing  $\mathcal{L}_{text}$ .
4: Use  $\mathcal{Z}$  to align the two modules by minimizing  $\mathcal{L}_{seq}$  (Equation 2) or  $\mathcal{L}_{tok}$  (Equation 3).
5: Discard the language module.
Output: The final speech module.
language module is kept fixed and only the speech module is updated.
To minimize the alignment loss, we randomly sample 10 hours of audio paired with its transcripts from the train-clean-360 subset, of which the speech portion is used to pre-train the speech module (§ 3.1). In practice, before minimizing the alignment loss, we find it beneficial to train (i.e., minimize  $\mathcal{L}_{text}$ ) the language module initialized with BERT<sub>BASE</sub> with the 10-hour LibriSpeech transcripts with the MLM task. This step allows the model to adapt to the speech domain and facilitates the following alignment task.
We summarize the complete procedure of pretraining SPLAT in Algorithm 1. After pre-training, the language module is discarded and only the speech module is used in downstream tasks.
We include a number of strong baselines from recent literature for each downstream task (Lugosch et al., 2019; Duran and Battle, 2018; Ghosal et al., 2018; Chuang et al., 2020). We also compare with another speech-language joint pre-training framework (Denisov and Vu, 2020). For each baseline, the reported performance is achieved by system that either uses similar or more amounts of data than our model.
To verify the effectiveness of each component in SPLAT, we experiment with the following variants of it, including whether to pre-train the model,
Table 1: Variants of SPLAT. An  $\pmb{x}$  indicates that the variant does not incorporate this step during pre-training. The step numbers correspond to those listed in Algorithm 1.
Model variant	Step 1. Pre-train speech module	Step 2. Pre-train language module	Step 3. Adapt language module before alignment	Step 4. Type of alignment loss
SPLAT-Scratch	X	X	X	X
SPLAT-Speech	✓	X	X	X
SPLAT-Seq	✓	✓	X	Lseq
SPLAT-Seq-MLM	✓	✓	✓	Lseq
SPLAT-Tok	✓	✓	X	Ltok
SPLAT-Tok-MLM	✓	✓	✓	Ltok
Table 2: Summary of SLU datasets. For the rows of Train, Validation, and Test, the numbers indicate the number of utterances in the split.
Task	Intent detection	Dialog act classification	Spoken sentiment analysis	Spoken question answering
Dataset	FSC	SwBD	CMU-MOSEI	Spoken SQuAD
Num. of classes	31	42	7	-
Train/val/test	23.1k/3.1k/3.8k	97.8k/8.6k/2.5k	16.2k/1.8k/4.6k	35.1k/2.0k/5.4k
whether to use the language module and which alignment task to apply. Table 1 summarizes the considered model variants.
- SPLAT-Scratch: No pre-training is conducted at all. Speech module is trained from scratch on downstream tasks.
- SPLAT-Speech: Only the speech module is pre-trained. Language module and alignment loss are not incorporated.
- SPLAT-Seq: SPLAT with sequence-level alignment loss  $\mathcal{L}_{seq}$ , but language module is not trained on LibriSpeech transcripts with MLM before alignment.
- SPLAT-Seq-MLM: SPLAT with sequence-level alignment loss  $\mathcal{L}_{seq}$ , and language module is trained on LibriSpeech transcripts with MLM before alignment.
- SPLAT-Tok: SPLAT with token-level alignment loss  $\mathcal{L}_{tok}$ , but language module is not trained on LibriSpeech transcripts with MLM before alignment.
- SPLAT-Tok-MLM: SPLAT with token-level alignment loss  $\mathcal{L}_{tok}$ , and language module is trained on LibriSpeech transcripts with MLM before alignment.
The speech module of SPLAT is a 3-layer Transformer encoder where each layer has a hidden size
of 768 and 12 self-attention heads. The language module is directly initialized from the pre-trained  $\mathrm{BERT}_{\mathrm{BASE}}$  released by Devlin et al. (2019).
4.2 Downstream SLU Tasks
We evaluate our model on four different SLU applications: intent detection, dialog act classification, spoken sentiment analysis, and spoken question answering. The first three belong to multi-class classification tasks, and the last one is a span prediction problem, which will be described in more detail below. Table 2 summarizes the used dataset for each application. For all datasets, we use 80-dimensional log Mel spectrograms as input acoustic features as in the pre-training stage.