SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks
Year: 2023

Abstract:
Spoken language understanding (SLU) tasks have been studied for many decades in the speech research community, but have not received as much attention as lower-level tasks like speech and speaker recognition. In this work, we introduce several new annotated SLU benchmark tasks based on freely available speech data, which complement existing benchmarks and address gaps in the SLU evaluation landscape. We contribute four tasks: question answering and summarization involve inference over longer speech sequences; named entity localization addresses the speech-specific task of locating the targeted content in the signal; dialog act classification identifies the function of a given speech utterance. In order to facilitate the development of SLU models that leverage the success of pre-trained speech representations, we will release a new benchmark suite, including for each task (i) curated annotations for a relatively small fine-tuning set, (ii) reproducible pipeline (speech recognizer + text model) and end-to-end baseline models and evaluation metrics, (iii) baseline model performance in various types of systems for easy comparisons. We present the details of data collection and annotation and the performance of the baseline models. We also analyze the sensitivity of pipeline models’ performance to the speech recognition accuracy, using more than 20 publicly available speech recognition models.

heading: 1 Introduction
text: Spoken language understanding (SLU) tasks involve inferring the linguistic structure or semantic meaning of a speech signal beyond its text transcript. We use this term broadly to include any natural language processing (NLP) task applied to speech, and tasks that involve linguistic understanding but also localization in the signal of relevant segments or producing speech as output. SLU
∗Core contributors in alphabetical order
has been an active area throughout the history of speech research (Hemphill et al., 1990; Calhoun et al., 2010; Busso et al., 2008; Zadeh et al., 2018; Chen et al., 2020a; Cohn et al., 2019; Yadav et al., 2020; Martinez-Lucas et al., 2020). However, compared to "lower-level" tasks like automatic speech recognition (ASR) and speaker identification, SLU has received much less attention and resources, and specifically there are much fewer benchmarks with freely available data.
SLU tasks can in principle be addressed via a pipeline approach — using ASR to map speech to text and an NLP (text) model to map text to the desired output. The alternative is an end-toend (E2E) model, which maps directly from the input speech to the target output. While pipeline approaches can take advantage of existing strong ASR and NLP models, E2E models can be more efficient at inference time, can avoid ASR error propagation, and can directly use aspects of the speech signal beyond the text that are useful for the task (e.g., prosody) (Arora et al., 2022a; Chen et al., 2020b; Jurafsky et al., 1998; Tran et al., 2018). In addition, for tasks whose output includes speech segments or time spans, there is no direct combination of an ASR model and an NLP model that produces precisely the desired type of output. For some SLU tasks, the current state of the art is a pipeline model (Shon et al., 2022a; Lai et al., 2020), whereas for others E2E models are better (Pasad et al., 2021; Sharma et al., 2022; Wu et al., 2022b; Peng et al., 2022; Arora et al., 2022b; Shon et al., 2022b). In order to better understand the pros and cons of pipeline and E2E approaches, more public benchmarks are sorely needed.
While collecting large amounts of labeled speech data for many SLU tasks may be prohibitively costly, recent advances in pre-trained models (Baevski et al., 2020; Hsu et al., 2021; Chen et al., 2021; Wu et al., 2022a; Baevski et al., 2022; Lin et al., 2022b; Mohamed et al., 2022) make
8906
it feasible to use relatively small fine-tuning sets for each task. There have been several recent efforts to introduce new benchmark SLU tasks (Yang et al., 2021; Bastianelli et al., 2020; Feng et al., 2021; Evain et al., 2021; Arora et al., 2022b; Lugosch et al., 2021a; Shon et al., 2022a; Tomasello et al., 2022), most (but not all) using fairly small training sets of several hours to several dozens of hours of speech. Among them, the Spoken Language Understanding Evaluation (SLUE)1 (Shon et al., 2022a) motivated us since it pursues a natural speech, rather than a short command type of speech that is populated in other benchmarks. However, there are only two SLUE tasks (sentiment analysis and named entity recognition), thus more tasks with different complexities are needed to cover the diverse application of SLU.
We introduce SLUE Phase-2, a set of SLU tasks that complement the existing SLU datasets or benchmarks. The new tasks include dialog act classification (DAC), question answering (QA), summarization (SUMM), and named entity localization (NEL) , applied to English speech data. SLUE Phase-2 has several advantages compared to other recent work introduced in section 2: More diverse tasks: SLUE phase-2 not only include utterance or word-level classification task but also includes QA and SUMM task. More challenging tasks: The complexity of the task is influenced by the type of input and the type of output. SLUE phase-2 uses conversational or longer discourse speech as input. The type of output is not limited to labels or text, but also includes the speech span time stamp. New human annotation: A new annotation was collected by a human annotator. Human annotator validated an automatically-collected data if needed. Natural speech: We do not use synthesized speech. We only include conversational or considerably long discourse speech rather than short speech commands. CC license: Creative Common licensed dataset to give the best freedom of use.
For each task, we provide publicly available2
datasets, annotations, models, and code. We provide both pipeline and E2E baseline models and, for pipeline models, we use multiple ASR systems to analyze the effect of the ASR error rate on the final task performance.
1We refer to the original SLUE as "SLUE Phase-1." 2To be released.

heading: 2 Related work
text: SUPERB (Yang et al., 2021) aggregates several existing speech tasks mainly to evaluate frozen pre-trained speech models. It focuses on lowlevel tasks but also contains two SLU tasks — intent classification (from Fluent Speech Commands (Lugosch et al., 2019)) and slot filling (from SNIPS (Coucke et al., 2018)). However, the former is an easy task where many models have close to 100% accuracy, and the latter uses synthesized rather than natural speech. SLURP (Bastianelli et al., 2020) is a spoken version of a text dataset (Liu et al., 2019) where the authors hired workers to dictate the written conversations between humans and personal robot assistants. It includes three SLU tasks — scenario prediction, action prediction, and entity prediction. These tasks cannot be generalized as the nature of the short speech command. ASR-GLUE (Feng et al., 2021) is based on the well-known GLUE benchmark (Wang et al., 2018) where the authors hired people to speak the GLUE text . It includes five GLUE tasks and one additional task. However, ASR-GLUE contains only a test set; researchers must rely on other datasets for training. Timers and Such (Lugosch et al., 2021b) is a dataset of speech commands that involve numbers, designed for intent classification and slot filling that has limited use case. Spoken SQuAD (Lee et al., 2018) and Spoken CoQA (You et al., 2022) are synthesized speech versions of the text SQuAD (Rajpurkar et al., 2016) and CoQA (Reddy et al., 2019) datasets. NMSQA (Lin et al., 2022a) is a multispeaker spoken QA dataset whose test set contains natural speech but the train and validation sets are synthesized. Other well-known SLU datasets include ATIS (Hemphill et al., 1990) and Switchboard NXT (Calhoun et al., 2010), which have been used for tasks like intent and DAC, but the data is available under license constraints. Wu et al. (2020) published an open-sourced speech dataset; however, its dialog act annotation is not manually annotated but predicted using commercial API.
Speech summarization has gained interest over the past few years with tasks such as abstractive summarization of instructional How-2 videos (Sanabria et al., 2018) and TED Talks (Kano et al., 2021), but the raw audio for these tasks is not publicly available. Other corpora, such as the ICSI (Janin et al., 2003) and AMI (McCowan et al., 2005) meeting summarization corpora, contain rel-
atively less annotated data. Named entity localization (NEL) is a fairly new task. A similar task, audio de-identification (audio de-ID), has been introduced with annotations for conversational data from Switchboard and Fisher (Cohn et al., 2019; Baril et al., 2022), but these datasets are not free. Audio de-ID is a special case of NEL where the entities of interest are related to personal identifiers.
We focus on English speech-related work (most comparable with our work), but there are also ongoing efforts for other languages (Tomashenko et al., 2019; Evain et al., 2021).

heading: 3 SLUE Phase-2: Tasks and data
text: This section introduces the tasks and metrics in SLUE Phase-2. The SLUE phase-1 introduced the "SLUE score", a numerical summary of model performance across tasks. However, as we consider a more diverse set of tasks, using the same pretrained model for all tasks is difficult, and evaluation via a single SLUE score may discourage building systems for individual tasks. In SLUE Phase-2, therefore, we do not adopt the single SLUE score, and evaluate each task individually.

heading: 3.1 Tasks
text: We explore more diverse and complex tasks compared to SLUE phase-1. As an extension of NER task in SLUE, we describe the NEL task to predict the audio time-stamps of named entities. DAC is an utterance classification task within conversation interactions to predict dialog acts given input speech. We address two longer-range context tasks: QA and SUMM where the model takes a long sequence and utilizes context across the entire scope to answer questions or summarize speech respectively.

heading: 3.1.1 Dialog Act Classification (DAC)
text: DAC is the task of identifying the function of a given speech utterance in a dialog, such as question, statement or backchannel. It is an utterance-level multi-label multi-class classification task; that is, an utterance can have more than one class (function). We evaluate DAC using macro-averaged (unweighted) F1 score.

heading: 3.1.2 Question Answering (QA)
text: The goal of QA is to find the answer span in a spoken document given a spoken question. The answer span is denoted by the start and end frames of a short phrase in the document. We use the frame-level F1 (frame-F1) score (Chuang et al.,
2020) to evaluate the overlap between the predicted and the ground-truth answer spans.

heading: 3.1.3 Speech summarization (SUMM)
text: SUMM refers to the task of generating a text summary from a given speech input. SUMM is challenging as it requires a model to assimilate information across very long input contexts in order to identify essential information and paraphrase to obtain the abstractive summary of speech. We evaluate SUMM using ROUGE (Lin, 2004), METEOR (Denkowski and Lavie, 2014) and BERTScore (Zhang* et al., 2020).

heading: 3.1.4 Named Entity Localization (NEL)
text: The goal of NEL is to predict the start and end times of any named entities in a spoken utterance. NEL is related to named entity recognition (NER), but NER involves identifying entity phrases while NEL involves locating them in the audio. We evaluate performance via two F1 scores based on the overlap between the predicted and ground-truth time ranges: frame-F1, defined similarly to the QA frame-F1 measure; and word-F1, defined similarly to the de-identification metric of Cohn et al. (2019). The word-F1 score has a hyperparameter ρ ∈ [0, 1], which is the fraction of overlap between a ground-truth word segment and the predicted region needed to count the word as detected; ρ = 1 means a perfect match is required.

heading: 3.2 Datasets and annotation

heading: 3.2.1 SLUE-HVB for DAC
text: For the DAC task we adapt the Harper Valley Bank (HVB) spoken dialog corpus3 (Wu et al., 2020) of scripted consumer banking dialogs, simulated by 59 speakers. The data contains about 23 hours of audio from 1,446 conversations with transcriptions and metadata, as well as dialog act annotation. However, the original DAC annotation is automatic, without manual validation, and the set of dialog acts is simple and tailored to this corpus. We define a new set of acts and collect human annotations by professional annotators listening to the audio. Our new set of dialog acts (See Table 9 in Appendix for detail) is based on the well-known Switchboard NXT (Calhoun et al., 2010) dialog act set. Based on a pilot annotation, we remove several unneeded labels and merge others unnecessarily granular. Finally, we split the HVB data into fine-tune, dev, and test sets (Table 1). The intent of conversation is
3CC-BY-4.0 license
balanced along the splits. We exclude short audio clips (<210ms) and audio that contains no speech.

heading: 3.2.2 SLUE-SQA-5 for QA
text: Previous open-source English spoken QA datasets, including Spoken SQuAD (Lee et al., 2018), NMSQA (Lin et al., 2022a), Spoken-CoQA (You et al., 2022), do not have a large training set consisting of realistic human speech, so we propose a new spoken QA dataset, SLUE-SQA-5, whose fine-tune, dev, and test sets all consist of real speech data.
The text transcriptions of question-answer pairs in SLUE-SQA-5 are collected from five different text QA datasets: SQuAD4 (Rajpurkar et al., 2016), Natural Questions5 (NQ) (Kwiatkowski et al., 2019), TriviaQA6 (Joshi et al., 2017), WebQuestions7 (WQ) (Berant et al., 2013), and CuratedTREC 8 (TREC) (Baudiš and Šedivỳ, 2015). We gather the text questions from the training set of the five text QA datasets as our fine-tune set. For our dev and test sets, we first collect the questions from the dev set of SQuAD, NQ, TriviaQA, WQ and the test set of TREC, and then randomly split these questions into two subsets as our dev and test sets. To get the spoken version of the collected questions, we used Amazon Mechanical Turk (MTurk), a crowdsourcing platform with anonymous, nonexpert workers, to collect spoken questions read by human speakers. The collection details are shown in Section B.1 in the Appendix.
For the documents, to avoid the enormous cost of collecting spoken versions of long text documents, we search for off-the-shelf spoken documents relevant to each question as paired documents from the Spoken Wikipedia dataset 4 (Köhn et al., 2016), which includes 1.2k spoken Wikipedia articles from about 400 different real speakers. We split the articles in Spoken Wikipedia into about 37k spoken documents with duration of 40 seconds.
We adopt a similar procedure with Joshi et al.
4CC BY-SA 4.0 license 5CC BY-SA 3.0 license 6Apache License 2.0 7CC-BY 4.0 license 8Public Domain
(2017) to search for relevant documents to the questions with their transcripts automatically. The detailed search criteria and the final number of SLUESQA-5 questions from each source text QA dataset are in Section B.2 and Table 11 in the Appendix.
To ensure the evaluation quality, we also asked human annotators to pick 408 question-document pairs, in which the document provides enough clues to answer the question, from test data as the verified-test set. The data statistics of SLUE-SQA5 are in Table 2.

heading: 3.2.3 SLUE-TED for SUMM
text: Of the existing corpora for abstractive speech summarization, How-2 has been used in recent work (Sharma et al., 2022). However, raw audio is not publicly available for the entire corpus, and the task of summarization is relatively easy due to shorter videos and simple reference summaries. Therefore, we consider the more challenging task of generating abstracts and titles for TED Talks, whose audio is publicly available. The TEDSummary dataset was introduced by (Kano et al., 2021) and accompanied by a tool to crawl and download TED talk videos from the web 9 that may be used to recreate the TEDSummary corpus. However, the lack of information about the exact talks used in the corpus makes it difficult to reproduce their data selection. Based on their crawler, and more recent talks released on the TED website10, we introduce SLUE-TED, a re-designed corpus of summaries for TED Talks spanning the years until 2022.
We find that, on average, nearly 66% of words in the title and 57.4% of words in the abstract are present in the transcript of a given audio, suggesting that ASR pre-training can be useful to improve speech summarization performance. For benchmark evaluation, we randomly split this corpus into 80% finetune, 10% validation, and 10% test set as shown in Table 3. A detailed description of the dataset is available in the Appendix C.2.
9https://github.com/nttcslab-sp-admin/TEDSummary 10CC BY–NC–ND 4.0 license

heading: 3.2.4 SLUE-VoxPopuli for NEL
text: SLUE-VoxPopuli was published with NER annotations in SLUE (Shon et al., 2022a). We extend SLUE-VoxPopuli to NEL by adding word-level time stamps in the dev and test sets. We use the Montreal Forced Aligner (MFA) (McAuliffe et al., 2017) to obtain word-level time stamps, using MFA’s public English acoustic model (McAuliffe and Sonderegger, 2022). MFA is a standard tool that is commonly used by the community to obtain ground-truth forced alignments. We manually verify the MFA produced entity alignments for 188 utterances (20% of the utterances with entity tags) in dev set and conclude that the MFA output provides a reliable ground-truth. We share more details for the data annotation and verification procedure in Appendix D.1. Data statistics for the SLUENEL data are shown in Table 4. Note that we do not publish NEL annotations for the finetune set as we focus on re-purposing NER models for NEL, which we believe is a more realistic use-case; as is also common for the speech-to-text forced alignment models, such as MFA, to be trained without ground-truth alignments.

heading: 4 Experiments and results
text: In the SLUE Phase-1 baseline experiments, larger pre-trained models and LM shallow fusion consistently gave better performance compared to smaller pre-trained models and without LM shallow fusion. Thus, in this paper, we analyze how the ASR word error rate (WER) in pipeline models is correlated with SLU task performance, by using multiple off-the-shelf open-source ASR models, specifically NeMo models (Kuchaiev et al., 2019) and
Whisper (Radford et al., 2022). Additionally, we quantify the performance gain on WER and SLU tasks achieved by fine-tuning custom ASR models compared to using off-the-shelf ASR models.
In all experiments, we use the fine-tune set of the corresponding task to fine-tune pre-trained models, the dev set to pick the best model, and the test set to evaluate both E2E and pipeline baselines. In addition, we measure the performance of an "oracle" pipeline system that uses ground-truth transcripts instead of ASR output. Below, we use the base sized model when there are multiple variants of the pre-trained model.

heading: 4.1 DAC
text: Baseline models: We follow a similar setup to the sentiment analysis baseline models in SLUE Phase-1 with some differences due to the multilabel nature of DAC. For the E2E baseline, we start with a pre-trained speech model, specifically wav2vec2 (Baevski et al., 2020), and add a selfattention pooling layer and two fully connected layers (including the output layer), with a Sigmoid output activation for each of the 18 dialog act classes. Outputs that is higher/lower than a threshold of 0.5 are classified as positive/negative for the corresponding class. For the pipeline baselines, we use either the off-the-shelf ASR models or an ASR using DAC data fine-tuned wav2vec2, and fine-tune a DeBERTa (He et al., 2020) model for the text classification.
Results: Table 5 shows the baseline results, and Figure 1a shows the relationship between WER and F1 score of pipeline models for a variety of ASR models (the ones used in Table 5 and all other NeMo models). We observe a strong correlation between the WER and DAC Macro F1 score (Pearson coorelation coefficient = -0.9). As the off-theshelf ASR models perform well on conversational speech, fine-tuning the ASR model does not give a large improvement over the best NeMo model.

heading: 4.2 QA
text: Pipeline Approach: The pipeline QA system is composed of an ASR model and a text QA model predicting the start and end words of the answer span on the ASR output transcript.
We fine-tuned DeBERTa with the ground-truth transcripts of the SLUE-SQA-5 fine-tune set to get the text QA model of all pipeline systems. Note that the DeBERTa text QA models in pipeline systems and the text QA models used for searching paired
documents (please refer to Section B.2) were finetuned on different datasets: the former were tuned on the SLUE-SQA-5 fine-tune set while the latter were tuned on the external SQuAD dataset.
When evaluating pipeline systems on the SLUESQA-5 dev and test sets, we used MFA to align ground-truth transcripts and ASR output transcripts to speech. The ground-truth answer words and the answer words predicted by the text QA model are converted to the time interval of the ground-truth and predicted answer span, which were then used to calculate the frame-F1 score.
E2E Approach: We used DUAL (Lin et al., 2022a) as the QA E2E approach (denoted as E2EDUAL). DUAL is composed of a wav2vec2-large model encoding speech waveforms, a k-means model converting wav2vec2 representations into cluster IDs, a Longformer model taking cluster IDs as input and predicting the start and end index of answer spans. We followed the training procedure in the DUAL paper except we used the k-means model of 500 clusters and fine-tuned its Longformer model for 45 epochs on the SLUESQA-5 fine-tune set.
Results: Table 6 shows the baseline results on the test and verified-test sets, and Figure 1b shows the relationship between document WER and frame-F1 on the test set of QA pipeline models. We observe a strong correlation (Pearson correlation coefficient=-0.89, p-value<0.01) between document WER and frame-F1. Pipeline-oracle significantly outperforms all the baseline models, and the performance gap is larger in the verified-test set, suggesting that there is room for improvement. Besides, the pipeline-w2v2 does not outperform the pipeline-nemo model, indicating that the fine-
tuned ASR model does not lead to better QA performance.

heading: 4.3 SUMM
text: Pipeline Approach: The oracle pipeline is constructed by using the ground truth transcript to train a text summarization model, and infer the most likely summary from the ground truth transcript. Then, we use different combinations of speech recognizers and text summarization models to build different pipeline models for speech summarization. For the pipeline baseline, we train ASR models on the TEDLIUM-3 (Hernandez et al., 2018) corpus using the ESPNet (Watanabe et al., 2018) toolkit. The ASR models consist of a conformer encoderdecoder architecture with pre-trained SSL representations as features (see Appendix C.1 for more details about our models). We also experiment with state-of-the-art off-the-shelf speech recognizers, including Whisper (Radford et al., 2022) and NeMo models. The resulting talk transcripts are very long, often exceeding 2048 tokens, requiring our text summarization models to be able to handle such long input sequences. Therefore, we use the Longformer Encoder-Decoder (LED-large) model (Beltagy et al., 2020), initialized using BART-large model (Lewis et al., 2019). We investigate training our text summarisation model on both ground truth and ASR transcripts.
E2E Approach: E2E speech summarization model is trained using the ESPNet (Watanabe et al., 2018) toolkit by first pre-training for speech recognition task on the TEDLIUM-3 corpus (Hernandez et al., 2018) and then fine-tuning on our SLUE-TED data for speech summarization task as described in (Sharma et al., 2022).
Results: Table 7 shows the performance for all baseline models on the test set (see Appendix C.3 for dev set performance). We observe that the performance of the pipeline system can be improved
by using a strong ASR model like Whisper. Further, we observe that the pipeline system performs slightly better when the text summarization model is fine-tuned on ASR transcripts. The pipeline models outperform the E2E system on ROUGE and METEOR, showing that the pipeline model aids in producing more accurate words. However, the end-to-end model does have a higher BERTScore, demonstrating the ability of the E2E model to produce semantically relevant summaries. All the baseline models perform worse than the pipeline-oracle model suggesting room for improvement.
To analyze the correlation between WER and the performance of the speech summarization task, we plot ROUGE-L scores in Figure 1c for various pipeline systems and a ground-truth transcriptbased text summarization model. We observe a strong correlation (Pearson correlation coefficient=0.9, p-value<0.01) between WER and ROUGE-L scores, suggesting that we can boost SUMM performance using a stronger ASR model.
To facilitate a better understanding of the performance of our E2E SUMM model, we analyze the percentage of exact matches in reference summary and predicted summaries for each POS tag. We observe that the majority of summarization errors occur because the model is not able to correctly generate the proper nouns in summary. A similar analysis on the percentage of exact matches for named entities shows that only 6.6% of entities in the reference summary were found in the predicted summary. Based on this analysis, we infer that the current speech summarization models struggle to correctly extract entities for the summary. (Full POS tags match available in Table 15 in the Appendix)

heading: 4.4 NEL
text: Baseline models: For NEL inference, we use the baseline NER models from Shon et al. (2022a). Both the E2E and ASR (within pipeline) models use wav2vec2 as the backbone and are trained with character-level connectionist temporal classification (CTC) (Graves et al., 2006). The text NER (within pipeline) model uses the DeBERTa as the backbone and is trained on ground-truth transcripts. Note that no dedicated model is trained for NEL. This is intentional: NER and NEL are related tasks and a realistic use case would require a single model that performs both tasks.
Inference: A CTC model produces a posterior
probability matrix, E ∈ RT×V , consisting of the posterior of each character in the vocabulary of size V for each of the T frames in the input audio. For ASR, the character vocabulary consists of the English alphabet, a word separator token “|”, and a blank token“ϵ”. For the E2E model, the vocabulary also includes special characters for the start and end of an entity phrase. We obtain a frame-level character sequence output via greedy decoding on E . The time stamps corresponding to “|” tokens in the output character sequence provide word-level start and end boundaries. As CTC is not trained with an explicit alignment signal, the word boundary tokens may not be a reliable indicator of the true time stamps, and we introduce two hyperparameters as a heuristic fix for possible mis-alignments: offset is a fixed duration by which we shift the time stamp predictions, and incl_blank ∈ {0, 1} denotes whether any trailing ϵ tokens are considered a part of the predicted entity segment.
In the pipeline approach, the predicted text from ASR is passed to a text NER model, and the time stamps for detected entities are extracted from the ASR’s E . For the E2E model, the time stamps corresponding to the entity start and end special characters are extracted directly from its E . An example is presented in Fig. 2.
Results: Table 8 presents the baseline results. The pipeline and E2E baselines have fairly similar frame-F1, but these approaches have complementary strengths as seen from their precision and
recall values (see Table 18, Appendix D.3). We also find that the off-the-shelf NeMo ASR model (pipeline-nemo) outperforms the dataset-specific ASR model (pipeline-w2v2).11
Figure 1d shows a scatter plot of NEL and WER scores for a variety of pipeline models. Although models with the lowest WER do have the best frame-F1, the overall correlation is not high. The NeMo models have different training objectives and model architectures, and we note that within each model class, the ASR and NEL metrics are much better correlated (see Figure 12, Appendix D.3). This suggests that model architecture and/or training objective play a significant role in alignment quality.12

heading: 5 Discussion
text: Among the baseline models, our pipeline models generally outperform their end-to-end counterparts. However, as shown in prior work (e.g., (Arora et al., 2022a; Pasad et al., 2021)), end-to-end models often have more room for improvement with careful and creative modeling ideas, and we hope that this new testbed helps spur such research.
In addition, the WER sensitivity analysis in Figure 1 suggests different strategies are needed for the
11More word-F1 results in Tab. 19 in Appendix D.4. 12The details of hyperparameter tuning and timestamp ex-
traction from NeMo models are in Appendix D.2.
pipeline system depending on the SLU task. For example, fine-tuned ASR (pipeline-w2v2) plays a significant role in the DAC task while the QA task is not, and ASR model architecture is critical for the NEL task while WER is more matter for DAC and SUMM tasks.

heading: 6 Conclusion
text: SLUE Phase-2, with four additional SLU tasks and high-quality annotation, enables a more comprehensive analysis of diverse SLU tasks than previously possible. Besides the task definitions and annotations, this work contributes multiple baselines and performance analysis using modern offthe-shelf ASR and text models. The baseline performance on all tasks is far from perfect, and the relative performance of different models differs across tasks, indicating that these tasks are ripe for additional work and analysis to push the boundary of SLU research.
Limitations
One limitation of this work is the lack of human performance scores on the new tasks. Although the baseline performance is far from perfect, and it seems quite likely that human performance is much better, this should be measured in future work. Another limitation is that it is unknown how much each task should benefit from access to the audio in addition to text; this could be measured in principle for humans, but again we leave this to future work.
Broader Impact and Ethics
Spoken language understanding benchmarks, like the ones we propose in this work, facilitate the development of technologies that may be particularly useful for speakers who are unable to read or write text and ultimately also for unwritten languages, where speech is the only form of communication. We hope that this work also spurs more collaboration across the fields of speech and natural language processing, both of which are needed to make progress in this area.
We ensured that the SLUE-SQA speech data collection from AMT was conducted with a higher wage (on average, US$10 per hour) than the US federal minimum wage. This wage includes compensation for the time spent on re-recording and addressing technical issues on the recording platform. We further took measures to ensure that our data collection and annotation process did not introduce any
potential biases in the SLUE Phase-2 benchmark. Specifically, for SLUE-SQA, we implemented an automatic check using the Google Speech-to-Text service. If the Word Error Rate (WER) exceeded 30%, workers were recommended to re-record the utterance. We chose a 30% WER threshold to identify and exclude empty or prematurely cut utterances. Our analysis showed that such violations were less than 8% of questions. Additionally, we personally listened to each recording and only discarded those where a significant portion of the content was missing. Recordings were accepted even if the WER exceeded 30%, ensuring that our dataset does not include any potential bias inherent in the automated speech-to-text service.
The DAC annotation in SLUE-HVB and verifiedtest set in SLUE-SQA data were done by ASAPP internal data labeling team. Everyone who participated in the annotation was an employee of ASAPP and conducted the work within the scope of their usual employment. Specifically, most of them have over 1 year of experience in speech and languagerelated data labeling and their education level is above a Master’s degree.

heading: Acknowledgements
text: We would like to thank Kyle Hager, and Molly Ruhl for their helpful comments and discussion from a linguistic perspective, and the whole ASAPP MLDL team members for high quality annotation. Part of this work used PSC Bridges2 and NCSA Delta through allocations CIS210014 and IRI120015 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.

heading: A DAC
text: A.1 Dialog act list Figure 4 shows the corelation between WER and F1 score on dev set. Table 10 shows the experiment result including dev set.
A.2 Annotation detail Figure 3 shows the annotation interface for DAC. Annotator could choose multiple acts per utterance. The annotator could listen to the corresponding speech segment for better judgment. Utterances are provided in chronologically by combining agent and caller channels. A single conversation was annotated by a single annotator. The total conversation was divided into 40 shards with evenly distributed intent of the conversation. A total of 5 annotators completed the annotation and we did not collect personal information such as the demographic or geographic background of the annotator.13
A.3 Model training details The E2E model fine-tuning was done with 2e-05 learning rate, 50,000 maximum update step and 2,800,000 maximum tokens for mini-batch. We use the macro-f1 score of dev set to choose the final model evaluation. We use single RTX GPU and took 2hours. Model training was done with 5 different random seed and reported median model. For pipeline system, wav2vec2 ASR model fine-tuning took 10 hours and DeBERTa NLP model took 3 hours using the same GPU. We followed the ASR and NLP fine-tuning script in SLUE-Toolkit. Reproducible baseline scripts will be released.
A.4 Additional results Figure 4 shows the WER and F1 score on dev set and it shows the same trend compared to test set presented in Figure 1a. Table 10 shows DAC task performance evaluation including dev and test set.
B QA
B.1 Spoken question collection details
To collection spoken questions in SLUE-SQA-5, we posted our own speech collection website to Mturk, asked each worker to read 50 questions and paid them 1 dollar, so the worker got 2 cents for reading one question. After the worker record their speech, our speech collection website uses Google Speech-to-Text service to transcribe the audio to text and calculate the WER. If the WER is higher than 30%, our website will notify the worker and suggest them recording again. In our manual check, we listened to every recording by ourselves and discarded a recording only when we found that a high portion of the content was missing; otherwise, we still accepted it even if the WER was over 30%. The interface of our speech collection website is shown in Figure 5.
B.2 Search criteria of SLUE-SQA-5 documents
When searching for the paired document to each question, we determined whether a document is relevant to a question by jointly considering (1) its rank among all documents in BM25 (Robertson et al., 2009) search, a common term-based retrieval algorithm that scores the relevance between texts via keyword matching, (2) its rank among all documents in semantic search with the sentence-transformers model14 (Reimers and Gurevych, 2019), a neural sentence-level semantic encoder pre-trained on 215M QA pairs from multiple datasets, and (3) word-F1 derived by passing the question and the document through three different text QA models151617 fine-tuned on SQuAD dataset. We discard a question if we found no relevant document for it.
In specific, for each question, we searched for documents that meet all the criteria listed below:
• The document transcript includes the answer string to the question.
• The document has one of the top-1000 highest BM25 scores with the question among all documents.
• The document has one of the top-100 highest relevance scores with the question among all documents in semantic search with the sentence-transformers model.
14https://huggingface.co/sentence-transformers/multi-qa-mpnet-base-dot-v1 15https://huggingface.co/Palak/microsoft_deberta-large_squad 16https://huggingface.co/deepset/deberta-v3-large-squad2 17https://huggingface.co/deepset/deberta-v3-base-squad2
• When we pass the question and document through the three pre-trained text QA models mentioned in Section 3.2.2, at least one model gets a non-zero word-F1 score. (This criterion is used for dev and test set questions only.)
If there exists a document that meet all the above criteria, we combine the document, question, and the question’s answer into a question-answer-document triplet. Otherwise, we consider the question unanswerable and discard it. Note that we limit the number of paired document per question to one. If we find multiple documents that meet the criteria, we will choose the one with highest relevance score in semantic search among them as the paired document.
B.3 Model training details The E2E-DUAL model is composed of a wav2vec2-large model encoding speech waveforms, a k-means model converting wav2vec2 layer representations into cluster IDs, and a Longformer model taking cluster IDs as input and predicting the start and end index of answer spans. We extract the representations of Librispeech (Panayotov et al., 2015) train-clean-100 set from the 22nd layer of the fixed wav2vec2-large model to train the k-means model. The k-means model is then used to convert the representations of SLUE-SQA-5 fine-tune set into discrete units, which are taken as the input to the Longformer model. We fine-tune Longformer with 1e-4 learning rate, 500 warmup steps and overall 128 batch size on 4 Tesla V100 gpus. It takes around 40 hours to fine-tune the Longformer model for 45 epochs. The total number of tuned parameters in DUAL, including the k-means model and Longformer part, is reported in Table 21.
For the pipeline system, we fine-tune the wav2vec2 ASR model with 1e-4 learning rate and 16 batch size for 10 epochs, and fine-tune the DeBERTa NLP model with 4e-5 learning rate, 100 warmup steps and 64 batch size for 10 epochs. Wav2vec2 ASR model fine-tuning takes 25 hours and DeBERTa NLP model takes 6.5 hours using one V100 gpu.
B.4 Additional results Figure 6 shows the relationship between the question WER and frame-F1 on the test set. We observe relatively weak correlation between question WER and frame-F1 compared to that between document WER and frame-F1.
Table 12 shows the QA performance on the dev set. Figure 7 shows the relationship between document WER and frame-F1 on the dev set and has the similar trend (Pearson correlation coefficient=-0.94, pvalue<0.01) compared to the test set in Figure 1b. Figure 8 shows the relationship between question WER and frame-F1 on the dev set. Similar to the test set, we observe relatively weak correlation between question WER and frame-F1 compared to that between document WER and frame-F1.

heading: C SUMM
text: C.1 Model details The ASR models consist of a conformer encoder-decoder architecture with pre-trained SSL representations like Hubert large (Hsu et al., 2021) and wav2vec2 large (Baevski et al., 2020) representations as features. Following prior work (Peng et al., 2022), a weighted sum of multiple hidden states of SSL models is utilized. Since the TED talks are very long, we break the audio into 10 second chunks, and infer the most likely transcript for each chunk independently. Then we concatenate the resulting transcripts from each audio chunk to obtain the talk transcript. ASR models were trained for nearly 23 hours on 4 v100 gpus.
The E2E speech summarization model has similar architecture as the ASR model of the pipeline baseline. Since the TED talks were too long to fit the entire speech input on a GPU, we use only the last hidden state of SSL model and trained our E2E model using only the first 30000 speech frames (600 seconds). E2E speech summarization model was trained for nearly 16 hours on 4 v100 gpus.
For Nemo conformer and squeezeformer models, the audio is too long to perform inference using a GPU, and hence we have to break audio input into 5-minute chunks and perform inference separately on each of these chunks.
C.2 Additional dataset details Table 13 summarizes the statistics of the dataset, and the distribution of ground truth transcript and summaries is shown in Figure 9. We observe that this dataset contains much longer audios and transcripts than prior works.
C.3 Additional results Table 14 shows the performance of all the models on the dev set. Figure 10 shows the correlation between WER and ROUGE-L scores on the dev set and has a similar trend to the one observed on test set in figure 1c. Table 15 show the percentage of exact matches in reference summary and predicted summaries for each POS tag on the test set. We further analyzed the performance of our E2E Summ model separately on abstract and title in summary and observed that the model performs slightly better at generating title (ROUGE-L:15.2, BERTScore:87.7) as compared to generating the abstract (ROUGEL:14.4, BERTScore:83.4). Table 16 provides example summaries generated by our baseline systems. We observe that pipeline models generate more accurate words while E2E model generates more semantically similar summaries to reference. However, both these models generate summaries that differ from references suggesting significant room for improvement.

heading: D Named entity localization
text: D.1 Annotation details
As described in Sec. 3.2.4, we use MFA to obtain ground-truth word-level alignments. When we run MFA, it fails to align twenty-six files across dev and test splits. On manual inspection we identify differences in audio utterance and the corresponding text transcript due to incorrect end-pointing for twenty-two of these files. These cases have contiguous words at the end of the transcript that are not a part of the audio utterance. Running MFA after removing these extra words from the transcripts fixes these cases. But, for seven of these files, at least one entity word is a part of the missing words and so, the time alignments don’t have all the entity phrases that are a part of the published SLUE-NER annotations. In the interest of utterance-level consistency between SLUE-NER and SLUE-NEL, we skip these files. For the remainder four of the twenty-six files that MFA fails to align, we manually add the word alignments using Praat software (Boersma and Weenink, 2009).
In order to check the validity of MFA produced alignments, we manually verify the entity alignments for 372 entity phrases across randomly chosen 188 utterances in dev split. This constitutes 20% of all entity phrases in the dev split and thus our analysis should be representative for the complete split. Our manual pass exposed 51 of 372 phrases to be misaligned and the nature of misalignment varied from a minor offset to being completely off. In order to quantify the effect of the identified misalignments on our evaluation metrics, we manually rectify the alignments for these 51 phrases and report the following scores for this representative set of 188 utterances: 1. The frame-F1 between rectified and original timestamps is 96%, 2. The relative difference in baseline model scores (evaluating models listed in Table 8) using these two
versions as ground-truths is <3%, 3. The general trend in baseline model scores is similar across models for the results using these two
versions as ground-truths. Thus, we conclude that the alignments produced by MFA are reliable for robustly comparing between different modeling approaches and can be used as ground-truth despite minor issues in the generated time-stamps. Additionally, we find that the faulty timestamps are a result of imperfect transcripts in VoxPopuli and not an issue with MFA. The imperfections in these transcripts are expected, since the data is originally curated with 20% character error rate threshold (Wang et al., 2021).
D.2 Hyperparameter details
NEL evaluation has two hyperparameters,offset and incl_blank. We evaluate the dev set on a range of offset values between -0.3 seconds and 0.3 seconds with an increment of 20 milliseconds. The incl_blank is a Boolean hyperparameter. The best hyperparameter values based on dev set performance are listed in Table 17.
The 34 NeMo models have one of the three types of decoding strategies – (i) character-level CTC, (ii) subword-level CTC, and (iii) subword-level RNN transducer (RNNT). The character-level CTC models are processed in the same way as the pipeline-w2v2 models, where the incl_blank denotes whether or not the ϵ tokens before and after the entity phrase, between the word separator tokens, are included in the entity time stamp. The subword-level CTC model vocabulary in the NeMo toolkit does not have a word separator token, and instead, the start of the word is characterized by an “_” prepended to a subword. So, the incl_blank denotes whether the trailing ϵ tokens, before the start of the next word, are included in the entity time stamp. The RNNT model class in the NeMo toolkit directly gives subword-level start times, so offset was the only relevant hyperparameter here.
D.3 Error analysis
Table 18 shows precision and recall values for the NEL models. The E2E model outperforms in precision (i.e, more predicted regions are named entities), whereas the pipeline model outperforms in recall. The mismatch in text NER’s training (ground-truth text) and inference (ASR output) could lead to higher false positives in the pipeline model.
Figure 12 shows the scatter plot between WER and F1 scores for NeMo, where the points are colorcoded for different base model types. We see that the NEL and ASR performance are correlated within a single model category.
Table 18: NEL task baseline precision and recall performance on dev set. *the best nemo model based on NEL frame-f1 score on dev is “stt_en_conformer_ctc_small"
System Speech model
Text model
frame-F1 word-F1 (ρ=1) word-F1 (ρ=0.8) word-F1 (ρ=0.5)
Prec. Recall Prec. Recall Prec. Recall Prec. Recall
pipeline-oracle x DeBERTa 91.7 92.8 92.4 94.7 92.4 94.7 92.4 94.7 pipeline-w2v2 wav2vec2 DeBERTa 57.8 78.8 70.4 46.4 71.1 74.1 68.5 84.9 E2E-w2v2 wav2vec2 x 81.0 51.7 71.8 19.5 83.8 55.0 83.2 63.2 pipeline-nemo best model* DeBERTa 69.2 83.2 82.4 56.4 83.7 83.1 79.7 88.1
0 5 10 15 20 25
WER
40
50
60
70
80
N E
R F
1
E2E-w2v2
pipeline-oracle
pipeline-w2v2
pipeline-nemo
Figure 11: NER task: WER and frame-F1 scores on dev set
D.4 Additional results Table 19 shows performance of NEL for dev and test sets across different thresholds for word-F1. For word-F1, relaxing the tolerance from ρ = 1 to ρ = 0.8 gives a major performance boost – up to 30% and 116% relative for pipeline and E2E models respectively.
Figure 13 shows the correlation between WER and frame-F1 on dev set. It follows a similar trend to test set (see Figure 1d).

heading: E Experiment detail
text: Table 20 shows NeMo model name list used in the experiment. Table 21 shows the number of parameters for model used in the experiment.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: limitations section in page 9.
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: in the abstract and section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: section 4.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. We don’t create any artifacts for distribution in this submission since a few of authors are prohibited to distribute dataset and source code anonymously. We will publish the dataset and reproducible code upon paper acceptance.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. We use the open-sourced dataset that already published and added additional annotation. Added annotation only is related to dialog act class and Document-Question pair validation. No new content is added in the original audio or original text dataset.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We don’t create any artifacts for distribution at this point.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. section 3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? in section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? in section 4 and appendix.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? in section 4 and appendix

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section 4.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? described in section 3 and appendix for detail.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? described at section 3 and more details at appendix.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? at section 3 and appendix

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? section 3

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Added detail in Broader Impact and Ethics section

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: In appendix.

heading: Future Work
text: Although the baseline performance is far from perfect, and it seems quite likely that human performance is much better, this should be measured in future work. Another limitation is that it is unknown how much each task should benefit from access to the audio in addition to text; this could be measured in principle for humans, but again we leave this to future work