SlowGuess/NAACL_2021_OCR huggingface dataset:
Intent detection We use the Fluent Speech Commands corpus (FSC) (Lugosch et al., 2019) for intent detection, where the goal is to correctly predict the intent of an input utterance. In this dataset, each utterance is annotated with three slots: action, object, and location, where each slot can take one of multiple values. The combination of slot values is defined as the intent of the utterance, and there are 31 unique intents in total. In this work we follow the original paper to formulate intent detection as a simple 31-class classification task.
Dialog act classification We use the NTX-format Switchboard corpus (SwDA) (Calhoun et al., 2010), a dialog corpus of 2-speaker conversations. The goal is to correctly classify an input
Table 3: Results on all downstream datasets. All numbers of our models are an average of three runs, of which variances are negligibly small and not included. The metric is classification accuracy for FSC, SwBD and CMU-MOSEI. The metric for Spoken SQuAD is Audio Overlapping Score (AOS).
Model	FSC	SwBD	CMU-MOSEI	Spoken SQuAD
Ours
SPLAT-Scratch	97.6	65.8	68.8	30.4
SPLAT-Speech	99.5	67.5	69.0	57.7
SPLAT-Seq	99.5	74.6	72.5	62.7
SPLAT-Seq-MLM	99.5	76.3	74.7	65.9
SPLAT-Tok	99.2	71.2	70.4	58.0
SPLAT-Tok-MLM	99.2	72.7	71.2	63.8
SPLAT-Seq-MLM 1-hour	99.5	75.8	65.3	65.3
Baselines
Lugosch et al. (2019)	98.8	-	-	-
Duran and Battle (2018)	-	75.5	-	-
Ghosal et al. (2018)	-	-	75.9	-
Chuang et al. (2020)	-	-	-	59.7
Denisov and Vu (2020)	95.5	60.2	-	-
utterance into one of the 42 dialog acts.
Spoken sentiment analysis We use the CMU-MOSEI dataset (Zadeh et al., 2018), where each utterance is annotated for a sentiment score on a  $[-3,3]$  Likert scale:  $[-3$ : highly negative, -2: negative, -1: weakly negative, 0: neutral, +1: weakly positive, +2: positive, +3: highly positive]. We treat the task as a 7-class classification problem. And we only use audio signals in the input data.
For the above three tasks, during fine-tuning, an MLP network with one hidden layer of 512 units is appended on top of the speech module. It converts the output representation of the first frame, i.e.,  $s_1$ , for class prediction. Both the pre-trained speech module and the randomly initialized MLP are finetuned on the training set for 10 epochs with a batch size of 64 and a fixed learning rate of 3e-4. We compute classification accuracy after each training epoch and pick the best-performing checkpoint on the validation set to report results on the test set.
Spoken question answering We use the Spoken SQuAD dataset (Li et al., 2018), which is augmented from SQuAD (Rajpurkar et al., 2016) for spoken question answering. The model is given an article in the form of speech and a question in the form of text. The goal is to predict a time span in the spoken article that answers the question. In other words, the model outputs an audio
segment extracted from spoken article as the answer. The model is evaluated by Audio Overlapping Score (AOS) (Li et al., 2018): the greater the overlap between the predicted span and the ground-truth answer span, the higher the score will be.
During fine-tuning, given a spoken article and a question in the text form, the pre-trained speech module extracts audio representations of the article and pass them to a randomly initialized 3-layer Transformer encoder along with the tokenized textual question as input. The Transformer then uses the self-attention mechanism to implicitly align elements of the input audio and textual features. For each time step of the audio input, the Transformer is trained to predict whether this is the start of the span with a simple logistic regression. A separate classifier is used for predicting the end of the span.
Table 3 shows the performance of models on all four downstream tasks. Each number from our model is an average over three runs. Based on the results, we make the following observations.
Firstly, compared with SPLAT-Scratch, all pretrained models achieve superior results, especially more than  $30\%$  gain on Spoken SQuAD, proving the effectiveness of pre-training.
Secondly, the inclusion of language module and the alignment task during pre-training is very beneficial. For instance, on CMU-MOSEI, SPLAT
$^{1}$ Li et al. (2018) used Google text-to-speech to generate the spoken version of the articles in SQuAD.
Figure 3: Performance on downstream tasks with varying training data sizes. All numbers are an average of three runs, of which variances are negligibly small and not included.
Training data size  $(\%)$
Seq-MLM outperforms SPLAT-Speech by  $5.7\%$ , and outperforms several baseline systems from recent literature. We argue that as SLU tasks require the model to interpret acoustic signals and their underlying semantics, the language module will guide the speech module towards a mutual understanding of both modalities via our alignment task.
Thirdly, updating the language module using MLM during pre-training is helpful. Although the language module has been initialized with BERT, adaptation to the speech domain can help with semantic understanding in the downstream task.
Types of alignment Comparing SPLAT-Seq against SPLAT-Tok, we find that sequence-level alignment outperforms token-level alignment on all four tasks, although the latter is supposed to learn more fine-grained multi-modal representations. We leave the investigations of reasons for such phenomenon and more advanced token-level alignment approaches for future work.
Low-resource scenario We experiment with a version of SPLAT that uses only 1 hour of transcribed speech randomly sampled from the LibriSpeech train-clean-360 subset for aligning speech and language modules, denoted as SPLAT-Seq-MLM 1-hour. The language module of SPLAT-Seq-MLM 1-hour—after being initialized with  $\mathrm{BERT}_{\mathrm{BASE}}$ —is trained on the 1-hour LibriSpeech transcripts before minimizing the alignment loss. It achieves comparable results with the best variant SPLAT-Seq-MLM: same accuracy on FSC,  $0.5\%$  less on SwBD, and  $0.6\%$  less on Spoken SQuAD. This shows that with a small amount of labeled speech data, our pre-training framework can achieve good results on downstream tasks.
5.2 Robustness to the size of downstream training data
As human labeling is time-consuming and labor-intensive, the amount of labeled training data for downstream tasks is often small and insufficient. In this section, we show that with effective pretraining, the model will be less dependent on the amount of downstream labeled data.
We randomly sample  $50\%$ ,  $10\%$ ,  $5\%$ , and  $1\%$  of the training data in the downstream tasks, and evaluate the performance of different variants of SPLAT when fine-tuned on the sampled data.
Figure 3 shows the performance on all four downstream tasks with varying training data sizes. We observe that among the variants, SPLAT-SeqMLM is least sensitive to training data sizes. For instance, in FSC, with only  $10\%$  of the training data, its accuracy only drops 0.4 points. In comparison, both SPLAT-Scratch and SPLAT-Speech drops about 10 points. And the gaps are in general larger when the size of training data further shrinks. Therefore, our proposed joint pre-training of speech and language modules can help the model quickly adapt to downstream tasks given a modest amount of training data.
5.3 The geometry of the speech latent space before and after alignment
So far we have empirically demonstrated the effectiveness of SPLAT for learning multi-modal speech-language representations that are useful in various SLU tasks. Here we further show that our sequence-level alignment loss (Equation 2) can help project two speech utterances that have similar textual embeddings to nearby points in the speech latent space.
Recall that we use the embedding of the first token/feature to represent an utterance and conduct sequence-level alignment (Equation 2). Sup
Table 4: Average cosine similarity between all pairs of speech embeddings  $(S_{avg})$ , and the average cosine similarity between a speech embedding  $s_1^{(p)}$  and that of an utterance whose textual embedding is closest to the corresponding textual embedding  $t_1^{(p)}$  ( $S_{closest}$ ).
Model	Savg	Sclosest
SPLAT-Speech	0.136	0.238
SPLAT-Seq	0.144	0.781
SPLAT-Seq-MLM	0.148	0.829
pose  $\pmb{t}_1^{(p)}$  and  $s_1^{(p)}$  correspond to the textual and speech embeddings of the first utterance by SPLAT and  $\pmb{t}_1^{(q)}$  and  $s_1^{(q)}$  correspond to the embeddings of the second utterance. Then, if  $\pmb{t}_1^{(p)} \approx \pmb{t}_1^{(q)}$ , our SPLAT model trained with the sequence-level alignment loss will produce  $s_1^{(p)} \approx s_1^{(q)}$ .
We use the dev-clean subset from the LibriSpeech corpus for the analysis. First, we compute the average pairwise cosine similarity between the utterances of all speech embeddings:
$$
S _ {a v g} = \frac {1}{K (K - 1) / 2} \sum_ {p = 2} ^ {K} \sum_ {q = 1} ^ {p - 1} \operatorname {c o s s i m} \left(\boldsymbol {s} _ {1} ^ {(p)}, \boldsymbol {s} _ {1} ^ {(q)}\right), \tag {4}
$$
where  $K$  is the number of utterances in dev-clean.
Next, for each utterance with its speech and textual embeddings denoted as  $s_1^{(p)}$  and  $t_1^{(p)}$  respectively, we first use  $t_1^{(p)}$  to retrieve the utterance with the most similar textual embedding  $t_1^{(q^*)}$ , i.e.,  $q^* = \operatorname{argmax}_{1 \leq q \leq K, q \neq p} \operatorname{cossim}(t_1^{(p)}, t_1^{(q)})$ . We then compute the cosine similarity between  $s_1^{(p)}$  and  $s_1^{(q^*)}$  and take the average of such value over all utterances in dev-clean:
$$
S _ {\text {c l o s e s t}} = \frac {1}{K} \sum_ {p = 1} ^ {K} \operatorname {c o s s i m} \left(\boldsymbol {s} _ {1} ^ {(p)}, \boldsymbol {s} _ {1} ^ {\left(q ^ {*}\right)}\right). \tag {5}
$$
We show the  $S_{avg}$  and  $S_{closest}$  of embeddings produced by SPLAT-Speech, SPLAT-Seq, and SPLAT-Seq-MLM in Table 4.
We see that  $S_{avg}$  is approximately the same for all model variants. However,  $S_{closest}$ , the average similarity between the speech embeddings of two linguistically similar utterances, increases from 0.238 to 0.781 after aligning the speech and language modules, and further increases to 0.829 after adapting the language module on LibriSpeech transcripts with MLM before the alignment. Overall, SPLAT can make a pair of semantically similar
utterances to have much closer speech embeddings, compared with other random pairs of utterances.
These results demonstrate that via an cross-modal alignment loss as simple as Equation 2, SPLAT can effectively transfer knowledge from the language module to the speech module to capture both acoustic and linguistic information of speech utterances.
Spoken language understanding (SLU) tasks require an understanding of the input audio signal and its underlying semantics. In this paper, we present a novel speech-language joint pre-training framework, SPLAT, to carry out both speech and language understanding tasks during pre-training. Besides a self-supervised training on the speech and language modules, we propose two methods to align the semantic representations from both modules using a modest amount of labeled speech data. The speech module can quickly adapt to downstream tasks and achieve superior results on various SLU datasets including intent detection, dialog act classification, spoken sentiment analysis, and spoken question answering. This joint pre-training also makes the model less sensitive to the amount of labeled training data in downstream domains.
For future work, we plan to integrate automatic speech recognition and natural language generation into our framework to achieve good results on spoken language generation tasks.
Alexei Baevski, Steffen Schneider, and Michael Auli. 2020a. vq-wav2vec: Self-supervised learning of discrete speech representations. In ICLR.
Alexei Baevski, Yuhao Zhou, Abdel-rahman Mohamed, and Michael Auli. 2020b. wav2vec 2.0: A framework for self-supervised learning of speech representations. In NeurIPS.
Aditya Bhargava, Asli Celikyilmaz, Dilek Hakkani-Tür, and Ruhi Sarikaya. 2013. Easy contextual intent prediction and slot detection. In ICASSP.
Sasha Calhoun, Jean Carletta, Jason Brenier, Neil Mayo, Dan Jurafsky, Mark Steedman, and David Beaver. 2010. The NXT-format Switchboard Corpus: A rich resource for investigating the syntax, semantics, pragmatics and prosody of dialogue. Language Resources and Evaluation, 44(4):387-419.
Yuan-Ping Chen, Ryan Price, and Srinivas Bangalore. 2018. Spoken language understanding without speech recognition. In ICASSP.
Yung-Sung Chuang, Chi-Liang Liu, and Hung-Yi Lee. 2020. SpeechBERT: Cross-modal pre-trained language model for end-to-end spoken question answering. In Interspeech.
Yu-An Chung and James Glass. 2020a. Generative pre-training for speech with autoregressive predictive coding. In ICASSP.
Yu-An Chung and James Glass. 2020b. Improved speech representations with multi-target autoregressive predictive coding. In ACL.
Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. 2019a. An unsupervised autoregressive model for speech representation learning. In *Interspeech*.
Yu-An Chung, Hao Tang, and James Glass. 2020. Vector-quantized autoregressive predictive coding. In Interspeech.
Yu-An Chung, Yuxuan Wang, Wei-Ning Hsu, Yu Zhang, and RJ Skerry-Ryan. 2019b. Semi-supervised training for improving data efficiency in end-to-end speech synthesis. In ICASSP.
Yu-An Chung, Wei-Hung Weng, Schrasing Tong, and James Glass. 2019c. Towards unsupervised speech-to-text translation. In ICASSP.
Pavel Denisov and Ngoc Thang Vu. 2020. Pretrained semantic speech embeddings for end-to-end spoken language understanding via cross-modal teacher-student learning. In Interspeech.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In NeurIPS.
Nathan Duran and Steve Battle. 2018. Probabilistic word association for dialogue act classification with recurrent neural networks. In EANN.
Deepanway Ghosal, Md Shad Akhtar, Dushyant Chauhan, Soujanya Poria, Asif Ekbal, and Pushpak Bhattacharyya. 2018. Contextual inter-modal attention for multi-modal sentiment analysis. In EMNLP.
Kyle Gorman, Jonathan Howell, and Michael Wagner. 2011. Prosodylab-aligner: A tool for forced alignment of laboratory speech. Canadian Acoustics, 39(3):192-193.
Parisa Haghani, Arun Narayanan, Michiel Bacchiani, Galen Chuang, Neeraj Gaur, Pedro Moreno, Rohit Prabhavalkar, Zhongdi Qu, and Austin Waters. 2018. From audio to semantics: Approaches to end-to-end spoken language understanding. In SLT.
Chia-Hsuan Lee, Shang-Ming Wang, Huan-Cheng Chang, and Hung-Yi Lee. 2018. ODSQA: Open-domain spoken question answering dataset. In SLT.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In ACL.
Chia-Hsuan Li, Szu-Lin Wu, Chi-Liang Liu, and Hung-Yi Lee. 2018. Spoken SQuAD: A study of mitigating the impact of speech recognition errors on listening comprehension. In Interspeech.
Shaoshi Ling and Yuzong Liu. 2020. DeCoAR 2.0: Deep contextualized acoustic representations with vector quantization. arXiv preprint arXiv:2012.06659.
Shaoshi Ling, Yuzong Liu, Julian Salazar, and Katrin Kirchhoff. 2020. Deep contextualized acoustic representations for semi-supervised speech recognition. In ICASSP.
Alexander H. Liu, Yu-An Chung, and James Glass. 2020a. Non-autoregressive predictive coding for learning speech representations from local dependencies. arXiv preprint arXiv:2011.00406.
Andy T. Liu, Shang-Wen Li, and Hung-Yi Lee. 2020b. TERA: Self-supervised learning of transformer encoder representation for speech. arXiv preprint arXiv:2007.06028.
Andy T. Liu, Shu-Wen Yang, Po-Han Chi, Po-Chun Hsu, and Hung-Yi Lee. 2020c. Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders. In ICASSP.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
Loren Lusosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua Bengio. 2019. Speech model pre-training for end-to-end spoken language understanding. In Interspeech.
Ha Nguyen, Fethi Bougares, Natalia Tomashenko, Yannick Esteve, and Laurent Besacier. 2020. Investigating self-supervised pre-training for end-to-end speech translation. In Interspeech.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. LibriSpeech: An ASR corpus based on public domain audio books. In ICASSP.
Daniel Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin Cubuk, and Quoc Le. 2019. SpecAugment: A simple data augmentation method for automatic speech recognition. In Interspeech.
Yao Qian, Rituja Ubale, Vikram Ramanaryanan, Patrick Lange, David Suendermann-Oeft, Keelan Evanini, and Eugene Tsuprun. 2017. Exploring ASR-free end-to-end modeling to improve spoken language understanding in a cloud-based dialog system. In ASRU.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Technical report, OpenAI.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD:  $100,000+$  questions for machine comprehension of text. In EMNLP.
Suman Ravuri and Andreas Stolcke. 2015. Recurrent neural network and LSTM models for lexical utterance classification. In *Interspeech*.
Morgane Rivière, Armand Joulin, Pierre-Emmanuel Mazaré, and Emmanuel Dupoux. 2020. Unsupervised pretraining transfers well across languages. In ICASSP.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019. wav2vec: Unsupervised pre-training for speech recognition. In Interspeech.
Dmitriy Serdyuk, Yongqiang Wang, Christian Fuegen, Anuj Kumar, Baiyang Liu, and Yoshua Bengio. 2018. Towards end-to-end spoken language understanding. In ICASSP.
Xingchen Song, Guangsen Wang, Zhiyong Wu, Yiheng Huang, Dan Su, Dong Yu, and Helen Meng. 2020. Speech-XLNet: Unsupervised acoustic model pretraining for self-attention networks. In Interspeech.
Gokhan Tur and Renato De Mori. 2011. Spoken language understanding: Systems for extracting semantic information from speech. John Wiley & Sons.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
Weiran Wang, Qingming Tang, and Karen Livescu. 2020. Unsupervised pre-training of bidirectional speech encoders via masked reconstruction. In ICASSP.
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In ACL.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Weinberger, and Yoav Artzi. 2020a. BERTScore: Evaluating text generation with BERT. In ICLR.
Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuaijiang Zhang, Xi Zhou, and Xiang Zhou. 2020b. Semantics-aware BERT for language understanding. In AAAI.
Chenguang Zhu, Ziyi Yang, Robert Gmyr, Michael Zeng, and Xuedong Huang. 2019. Make lead bias in your favor: A simple and effective method for news summarization. arXiv preprint arXiv:1912.11602.
Chenguang Zhu, Michael Zeng, and Xuedong Huang. 2018. SDNet: Contextualized attention-based deep network for conversational question answering. arXiv preprint arXiv:1812.03593.